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INTRODUCTION 


Although  many  complex  systems  are  operated  by  teams  rather  than  by  single 
operators,  the  cognitive  and  behavioral  processes  driving  effective  team  performance  in 
dynamic,  real-world  settings  are  not  well  understood  (Regian  &  Elliott,  1997).  To  address 
this  shortcoming,  the  Air  Force  initiated  the  New  World  Vistas  (NWV)  Cognitive 
Engineering  for  Teams  program.  Central  to  this  program  is  the  development  of  a  number 
of  synthetic  team  task  environments  (STTEs),  defined  as  computer-based  artificial  team 
tasks  designed  specifically  for  basic  research  purposes,  yet  modeled  after  real  world  tasks. 
These  STTEs  will  ultimately  be  used  in  laboratory  studies  aimed  at  investigating  the 
characteristics  of  team  behavior  and  cognition.  As  such,  STTEs  will  offer  a  number  of 
advantages  over  the  use  of  both  conventional  simulators  and  existing  artificial  tasks; 

•  Unlike  participants  in  simulator  studies,  participants  in  studies  using  STTEs  will  not 
have  to  be  domain  experts.  STTEs  will  be  simple  enough  that  naive  subjects  will  be 
able  to  become  task-proficient  in  a  matter  of  hours. 

•  Unlike  simulators,  STTEs  will  be  reconfigurable:  researchers  will  have  the  flexibility  to 
systematically  manipulate  important  task  characteristics  and  collect  data  reflecting  the 
effects  of  these  manipulations. 

•  Unlike  many  existing  artificial  tasks,  STTEs  will  be  ecologically  valid,  that  is,  care  will 
be  taken  to  ensure  that  conclusions  drawn  from  studies  performed  using  STTEs  will 
also  apply  to  the  real  world  task  on  which  the  STTE  is  modeled. 

However,  building  an  STTE  is  not  simply  a  matter  of  choosing  a  real  world  task 
and  producing  a  simplified  version  of  it.  While  STTEs  certainly  need  to  be  simpler  than 
their  analogous  real  world  tasks,  the  value  of  the  STTE  as  a  research  vehicle  depends 
heavily  on  the  way  in  which  simplification  is  achieved.  The  purpose  of  an  STTE  is  to  allow 
investigation  of  task  characteristics  that  drive  task  performance  in  the  real  world,  so  it  is 
important  that  critical  real  world  task  characteristics  be  preserved  in  STTEs. 

To  achieve  this,  one  must: 

•  Determine  the  important  characteristics  of  the  real  world  task:  the  major  features  and 
goals  of  the  task,  the  subtasks  that  enable  the  goals  to  be  attained,  the  decisions  and 
problems  facing  performers  of  the  ta;sk,  and  the  cues  and  information  used  to  support 
decision-making  and  problem  solving 

•  Build  the  STTE  to  share  these  characteristics 

For  example,  if  a  major  concern  for  Air  Traffic  Controllers  (ATCs)  is  scheduling 
runway  space  for  flights  arri\fing  simultaneously,  then  an  STTE  based  on  the  ATC  task 
should  require  subjects  to  solve  similar  scheduling  problems,  using  similar  cues  and 
information.  Conversely,  there  may  be  characteristics  of  the  real  world  task  that,  although 
superficially  striking,  are  not  important  determinants  of  performance  and  so  should  not  be 
preserved  in  an  STTE.  For  example,  once  a  runway  schedule  has  been  made,  real  world 
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ATCs  may  have  to  pass  it  on  to  other  personnel  by  executing  a  number  of  console 
operations.  Although  these  operations  may  appear  complex,  controllers  may  carry  them 
out  with  little  effort  due  to  such  procedures  being  highly  learned.  However,  if  the  STTE 
were  designed  such  that  passing  on  the  runway  schedule  also  required  performance  of  a 
complex  set  of  operations,  STTE  subjects  would  have  difficulty  performing  this  operation, 
possibly  making  errors  that  have  no  real  world  counterparts.  Therefore,  the  goal  should  be 
to  simplify  complex  operations  such  that  each  STTE  operation  makes  the  same  relative 
demand  of  STTE  subjects  as  the  analogous  real  world  operation  does  of  real  world 
operators. 

Cognitive  Task  Analysis  (CTA)  methods  are  used  to  analyze  the  performance  of 
tasks  that  have  large  cognitive  components:  that  is,  when  task  performance  is  highly 
dependent  on  mental  processes.  CTA  methods  go  beyond  traditional  task  analysis 
approaches,  aiming  to  describe  characteristics  of  tasks  such  as  workload,  decision-making 
strategies,  and  errors.  It  is  this  focus  that  makes  CTA  ideal  for  guiding  STTE  design: 
knowing  the  cognitive  characteristics  of  a  real  world  task,  an  STTE  can  be  designed  to 
have  the  same  cognitive  characteristics,  yet  be  behaviorally  much  simpler.  Accordingly, 
the  NWV  program  has  funded  our  group  to  perform  CTAs  in  the  service  of  STTE 
development,  simultaneously  advancing  CTA  methods  to  cope  with  dynamic,  real-time 
team  tasks.  In  this  report,  we  present  the  results  of  a  CTA  of  the  AWACS  Weapons 
Director  task.  The  Weapons  Director  task  is  the  real  world  counterpart  of  “SynTEAM” 
(Synthetic  Team  Effectiveness  Assessment  and  Modeling),  an  STTE  in  development. 


Domain  Overview 

The  E3  Airborne  Warning  and  Control  Systems  (AWACS)  aircraft  carries  both 
surveillance  personnel  and  Weapons  Director  (WD)  teams.  WDs  are  responsible  for  the 
command  and  control  of  aircraft  within  a  particular  area.  In  an  existing  cognitive  task 
analysis  of  AWACS  WDs,  Klinger,  Andriole,  Militello,  Adelman,  and  Klein  (1993) 
describe  the  WD  task: 

“The  WD  position  can  be  likened  to  that  of  an  Air  Traffic  Controller  in  the 
sky,  with  some  important  differences:  commercial  aircraft  seldom  shoot  at 
one  another,  the  Air  Traffic  Controller  never  needs  to  monitor  an  airborne 
track  in  order  to  determine  intent;  Air  Traffic  Controllers  are  seldom  in 
danger  of  being  shot  down  (they  are  not  flying  in  the  sky  with  the  aircraft 
they  are  controlling);  and  they  do  not  need  to  worry  about  rules  of 
engagement.  A  WD  must  contend  with  all  of  these  and  rhore.”  (p.  3) 

Another  important  difference  between  Air  Traffic  Controllers  and  WDs  is  that 
WDs  are  primarily  concerned  with  directing  aircraft  toward  each  other,  as  in  the  case  of 
orchestrating^ an  intercept  or  an  aerial  refueling,  rather  than  exclusively  trying  to  keep 
aircraft  apart*.  Performing  this  task  requires  great  precision  and  attention,  and  guiding 


Although  WDs  also  do  this:  they  deconflict”  aircraft  so  they  don’t  inadvertently  fly  into  one  another. 
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aircraft  towards  one  another  makes  great  perceptual  demands  on  the  operator,  more  so 
than  keeping  separation  between  aircraft  or  directing  an  aircraft  to  a  stationary  runway  on 
the  ground.  A  small  mistake  on  the  part  of  a  WD  could  make  the  difference  between 
making  a  successful  intercept  and  allowing  a  hostile  plane  to  penetrate  friendly  airspace. 
Further,  WDs  have  to  contend  with  different  mission  objectives:  for  example,  a  mission 
might  involve  patrolling  a  quiet  airspace,  reconnaissance,  launching  a  defensive  counter  air 
strike  in  response  to  an  enemy  attack,  or  controlling  an  offensive  strike  package 
penetrating  into  enemy  territory,  as  well  as  non-combat  missions  such  as  search-and- 
rescue.  To  add  further  complications,  within  a  given  mission  WDs  have  to  perform  a  range 
of  different  tasks,  such  as  monitoring  aircraft  launches  and  landings,  monitoring  enemy 
aircraft,  running  intercepts  (executed  in  different  ways  depending  on  the  current  rules  of 
engagement  and/or  the  relative  strengths  of  the  aircraft  involved),  overseeing  rescue 
missions  for  downed  pilots,  and  mid-air  refueling  of  aircraft.  Performing  these  tasks 
requires  tracking  an  enormous  amount  of  information,  such  as  the  position,  heading, 
altitude,  and  speed  of  both  fnendly  and  hostile  aircraft,  and  the  fuel  and  armament  status 
of  friendly  aircraft,  as  well  as  maintaining  awareness  of  the  current  Rules  of  Engagement 
(ROE).  Contending  with  all  this  information,  WDs  have  to  manage  the  deployment  of 
their  often-limited  assets  to  deal  effectively  with  all  that  is  going  on  in  their  airspace,  or 
Area  of  Responsibility  (AOR).  In  such  a  highly  d5maniic  and  fast-paced  environment,  this 
is  an  extremely  challenging  task. 


AOR  3 


AOR  2 


AORl 


Ground 

Control 


Fieure  1:  Illustration  of  how  multiple  AW  ACS  aircraft  work  in  tandem 


The  workload  involved  in  directing  a  whole  mission  is  clearly  too  much  for  one 
person,  and  that  is  why  WDs  work  in  teams.  Team  size  varies  according  to  the  projected 
demands  of  the  mission.  A  typical  set-up  for  a  routine  patrol  mission  will  have  3  WDs, 
while  a  more  involved  strike  mission  might  have  6  WDs.  On  the  AW ACS  aircraft,  WDs  sit 
at  individual  consoles  arranged  in  rows  of  three.  Within  a  team,  each  WD  takes  on 
different  responsibilities.  Responsibilities  might  be  divided  according  to  geography,  with 
each  WD  controlling  operations  in  a  different  portion  of  the  airspace  (or  “lane”),  or 


3 


according  to  function,  with  each  WD  responsible  for  different  functions  (such  as  control  of 
airborne  tankers  and  refueling  operations,  responsibility  for  aircraft  check-in,  or  control  of 
a  strike  package)  within  the  total  AOR.  For  missions  involving  a  large  number  of  forces  to 
be  controlled,  two  or  more  WDs  might  be  assigned  to  one  functional  role;  for  example,  in 
a  mission  involving  a  large  strike  force  there  may  be  a  main  strike  controller  and  one  or 
two  “assist”  controllers.  Currently,  the  functional  division  of  labor  is  far  more  common.  In 
situations  in  which  there  is  a  large  quantity  of  airspace  to  be  controlled,  or  when  it  is 
advantageous  to  divide  the  airspace  into  discrete  regions,  a  number  of  separate  AWACS 
aircraft  might  be  used,  each  having  on  board  their  own  team  of  WDs  responsible  for  their 
own  AOR,  and  each  answerable  to  Ground  Control  (Figure  1). 


Figure  2;  Illustration  of  the  chain  of  command  on  a  single  AWACS  aircraft 


Ground 

Control 


Each  WD  team  falls  under  the  leadership  of  a  Senior  Director  (SD),  who  sits  at  a 
console  located  behind  those  of  the  WDs.  SDs  are  qualified  WDs  who  have  received  extra 
training,  but  who  are  not  necessarily  more  experienced  than  the  WDs  on  their  teams  since 
only  officers  can  be  SDs.  The  SD  monitors  the  WD  team,  facilitating  coordination 
between  WDs  and  generally  making  sure  that  the  mission  stays  on  track.  SDs  often  use  a 
Socratic  method”  to  ensure  that  WDs  maintain  their  situation  awareness.  For  example, 
the  question  “what  is  that  in  the  North-West  comer  of  your  lane?”  might  not  really  be  a 
request  for  information,  but  a  reminder  to  the  WD  that  there  is  an  unnoticed  hostile 
coming  into  the  lane  that  needs  to  be  dealt  with.  WDs  often  face  decisions  that  are  outside 
their  authority  (in  fact,  most  decisions  at  the  strategic  level  are  beyond  WDs’  authority). 
WDs  work  through  these  decisions  with  their  SD,  who  can  act  autonomously  or  confer 
with  the  chain  of  command  (i.e.  the  Mission  Commander  [MC]  and  then  Ground  Control 
as  shown  in  Figure  2). 


All  WDs,  and  the  SD,  sit  at  identical  consoles  consisting  of  a  radar  screen 
(  scope  ),  a  trackball,  an  alphanumeric  keyboard,  a  number  of  specialized  switch  panels, 
and  a  radio.  WDs  receive  information  fi-om  both  their  scope  and  the  radio,  and 
experienced  WDs  place  roughly  equal  weight  on  both  sources.  The  main  feature  of  the 
scope  is  a  graphical  display  of  the  AOR,  showing  data  from  AWACS  radar  returns 
superimposed  over  an  outline  map  of  the  ground  territory.  The  map  shows  basic 
information  such  as  land/sea  borders,  international  borders,  cities,  airfields,  and  missile 
sites. 
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The  radar  provides  the  current  positions,  headings,  altitudes,  and  speeds  of 
aircraft  in  the  AOR,  as  well  as  any  identification  modes  and  codes  that  aircraft  are 
transmitting  (“squawking”).  Radar  returns  are  known  as  “tracks”.  Track  information  is 
updated  with  every  radar  sweep,  or  every  12  seconds.  However,  not  all  of  the  available 
information  is  usually  displayed  at  once,  depending  on  which  options  each  WD  has  chosen 
in  setting  up  his^  scope.  Typically,  the  system  displays  current  track  positions  as  dots,  with 
headings  signified  by  the  direction  of  short  lines  originating  at  each  dot,  and  track 
“histories”  displayed  as  a  series  of  blinking  dots  showing  each  track’s  position  at  the  last 
three  radar  sweeps  (i.e.  the  last  36  seconds).  The  histories  provide  WDs  with  a 
representation  of  aircraft  movement,  making  it  easier  to  see  when  tracks  change  heading. 
The  color  of  the  dot  conveys  information  about  the  identification  modes  and  codes  that 
the  aircraft  is  transmitting.  A  green  dot  means  that  the  aircraft  is  transmitting  an 
identifiable  code  (and  so,  barring  any  counter-measures,  is  probably  a  “friendly”),  a  yellow 
dot  means  that  it  is  not  transmitting  identifiable  codes  (and  so  is  probably  not  a  friendly). 

Tracks  are  fiirther  identified  by  “tagging”  them  with  a  symbol  and  track  number, 
or  “symbology”.  The  symbology  identifies  the  aircraft  as  a  bogey  (unknown),  bandit 
(identified  enemy),  or  friendly.  WDs  are  typically  responsible  for  identifying  and  tagging 
friendly  aircraft  (those  transmitting  identifiable  codes),  while  identification  of  unknown 
aircraft  is  usually  done  by  the  surveillance  team,  also  working  from  the  AW  ACS  aircraft. 

WDs  also  use  their  consoles  to  record  and  keep  track  of  mission  progress.  For 
example,  when  a  command  is  issued  to  an  aircraft,  it  should  ideally  be  reflected  by  an  input 
to  the  system,  usually  achieved  via  one  of  the  80  “switch  actions”  located  on  a  panel  to  the 
right  of  each  scope.  Completed  switch  actions  are  propagated  through  the  whole  system 
and  are  reflected  in  the  symbology  and  information  available  at  each  console.  Thus  if  a 
WD  sends  a  fighter  to  intercept  a  hostile  aircraft,  this  should  be  reflected  with  a  “commit” 
switch  action,  and  the  aircraft’s  commit  status  is  shown  as  part  of  its  symbology.  Since 
this  information  is  available  to  all  the  other  WDs  (as  well  as  the  SD,  the  MC,  and  Ground 
Control,  all  of  who  may  be  looking  at  an  AW  ACS  radar  screen),  it  is  important  that  the 
system  be  kept  current. 

Of  the  80  switch  actions  at  a  WD’s  disposal,  some  are  intended  for 
communication  with  the  AW ACS  system,  some  determine  display  characteristics  for 
individual  scopes  (such  as  zoom  and  declutter  functions),  while  others  activate  tools  to  aid 
WDs  in  decision  making,  such  as  tools  to  calculate  intercept  geometry.  Information  not 
displayed  on  the  map  display  can  also  be  displayed  numerically  as  a  tabular  display  (TD). 
TDs  are  usually  displayed  on  the  scope  just  below  the  map,  although  it  is  possible  to 
display  full-screen  TDs  and  toggle  between  these  and  the  map  display.  Each  WD  is  free  to 
set  his  display  options  independently,  so  although  all  WDs  have  access  to  the  same 
information  their  individual  scopes  may  look  very  different  at  any  given  time. 


^  Throughout  this  report  we  have  used  male  pronouns  to  refer  to  WDs  and  pilots,  as  there  are  no  gender- 
neutral  alternatives  that  do  not  render  the  text  awkward. 
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As  for  the  radio,  the  channels  on  all  WDs’  radios  are  tuned  to  receive  the  same 
frequencies.  The  Gruard  channel  is  an  international  frequency  used  for  general  purpose  and 
emergency  communication  between  aircraft.  Four  intercom  channels,  known  as  Net  1 
(flight  deck).  Net  2  (WDs  and  SD),  Net  3  (surveillance),  and  Maintenance,  are  used  for 
internal  communication  aboard  the  AW  ACS  aircraft  itself  WDs  sometimes  talk  about 
“Net  4”,  but  this  is  merely  WD  jargon  for  face-to-face  verbal  communication. 

The  remaining  four  channels,  known  as  A,  B,  C,  and  D,  are  used  for 
communicating  with  other  aircraft.  The  frequencies  of  these  channels  are  changed  for  each 
mission.  Although  each  WDs’  radio  is  tuned  to  the  same  four  mission  frequencies,  the 
frequencies  are  configured  differently  on  each  radio.  Each  WD’s  radio  is  configured  such 
that  the  aircraft  with  which  he  is  most  likely  to  communicate  are  broadcasting  on  his  “A” 
channel.  For  example,  a  WD  controlling  patrol  fighters  would  be  able  to  communicate 
with  all  of  his  fighters  on  Channel  A.  For  the  WD  controlling  tankers  in  the  same  mission, 
Channel  A  would  be  tuned  to  a  different  frequency,  on  which  tankers  would  communicate. 
However,  the  WD  controlling  the  fighters  would  also  be  able  to  communicate  with  the 
tankers,  since  either  his  B,  C,  or  D  channel  would  be  tuned  to  receive  the  tanker 
frequency.  A  WD  controlling  aircraft  check-in  would  receive  a  special  check-in  frequency 
on  Channel  A,  on  which  all  aircraft  broadcast  as  they  take  off.  The  WDs  controlling 
fighters  and  tankers  would  have  access  to  this  frequency  on  their  Channels  B,  C,  or  D, 
while  the  WD  performing  check-in  duties  would  be  able  to  access  the  fighter  and  tanker 
frequencies  on  his  other  channels.  It  is  possible  for  a  WD  to  pass  control  of  an  aircraft  to 
another  WD,  and  when  this  is  done  the  pilot  is  “pushed”  to  the  new  WD’s  Channel  A 
frequency. 

WDs  can  listen  to  any  combination  of  frequencies  at  one  time.  They  control  which 
of  the  channels  they  hear  by  adjusting  the  volumes:  a  common  listening  strategy  is  to  turn 
up  the  volume  on  the  most  relevant  channels  (typically  Channel  A,  Net  2,  and  Guard)  and 
modulate  the  volume  on  all  others,  listening  to  some  at  a  reduced  volume  and  turning 
others  off  completely.  To  speak  over  a  particular  channel,  the  WD  pulls  out  the  button  for 
that  channel,  then  presses  on  a  pedal  while  he  talks.  It  is  possible  to  talk  on  more  than  one 
channel  at  one  time  by  pulling  out  more  than  one  button. 

Using  the  console,  WDs  communicate  vvdth  their  pilots,  and  among  themselves,  to 
successfully  complete  a  range  of  different  types  of  missions  in  which  many  different  kinds 
of  unexpected  event  might  occur.  However,  WDs  do  this  with  almost  no  training  in 
teamwork  or  team  interaction.  Prior  to  being  certified  “mission  ready”;  the  only 
experience  a  WD  has  had  with  a  team  is  simulator  practice,  with  no  overt  instruction  on 
teamwork. 

For  a  typical  mission,  WDs  start  with  a  comprehensive  briefing.  Briefings  may  last 
a  number  of  hours,  and  during  this  time,  WDs  review  the  mission  objectives,  the  schedule 
of  aircraft  that  will  be  flying,  known  as  the  Air  Tasking  Order  (ATO),  and  the  Rules  of 
Engagement  (ROE),  which  define  how  aircraft  should  behave  in  case  of  contact  with 
unknown  or  enemy  aircraft.  The  ROE  are  liable  to  change  during  a  mission  (e  g.,  aircraft 
may  be  put  on  a  higher  state  of  alert),  and  WDs  discuss  this  possibility  and  how  they  will 
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deal  with  it.  WDs’  roles  and  responsibilities  are  clearly  defined  during  the  briefing, 
dictating  how  they  will  communicate  and  coordinate,  and  WDs  work  out  among 
themselves  how  they  will  communicate  and  coordinate  with  each  other  in  special 
circumstances  by  way  of  verbal  “contracts”.  Most  missions  that  WDs  fly  are  routine  and 
uneventful,  with  aircraft  flying  more  or  less  according  to  the  ATO.  These  types  of  mission 
pose  very  few  problems  for  WDs.  When  non-scheduled  and  unexpected  events  occur, 
such  as  encounters  with  enemy  aircraft,  the  WD  task  becomes  very  challenging,  and 
teamwork  becomes  critical,  as  WDs  become  more  overloaded  and  events  deviate  further 
from  a  priori  plans. 


This  Study 

Our  goal  in  this  study  is  to  describe  the  important  characteristics  of  the  WD  team 
task  and  to  provide  preliminary  specifications  for  SynTEAM  design.  To  achieve  this,  we 
used  a  number  of  different  interviewing  methods  that  converge  on  a  common  task 
description.  “General”  interviews  were  semi-structured  interviews  using  questions  derived 
from  the  researchers’  initial  domain  analyses  using  documentation,  wdeotapes,  and 
informal  discussions  with  a  subject  matter  expert  (SME).  The  purpose  of  the  general 
interviews  was  to  elicit  broad,  context-free  knowledge  and  information  about  the  WD  task 
in  order  to  better  guide  the  more  detailed  analyses  planned  for  the  future,  and  also  to 
capture  the  stable  domain  knowledge  underlying  task  performance.  “Critical  incident” 
interviews  (Klein,  Calderwood,  &  MacGregor,  1989)  were  usually  conducted  in  the  same 
session  as  general  interviews,  and  were  used  to  sample  real  decisions,  procedures  and 
strategies  that  underlie  actual  task  performance.  Finally,  “quasi-performance”  interviews 
used  an  adaptation  of  PARI  (Hall,  Gott,  and  Pokomy,  1995)  in  order  to  elicit  detailed  task 
knowledge.  The  main  strength  of  performance  interviews  is  that  knowledge  is  elicited  in 
the  context  of  actual  task  performance,  either  real  or  simulated,  thus  targeting  “knowledge 
in  action”.  While  critical  incident  interviews  can  also  yield  detailed  information  elicited  in 
the  context  of  task  performance,  performance  interviews  have  a  number  of  unique 
features: 

•  Data  are  collected  in  the  context  of  a  full  array  of  environmental  cues,  providing  the 
interviewee  with  ecologically- valid  “prompts”.  Data  are  thus  be  more  complete  than 
data  collected  retrospectively,  minimizing  the  chance  of  missing  important  information 
due  to  failures  of  memory. 

•  Since  the  researcher  is  present  during  task  performance,  he/she  can  probe  the 
interviewee  about  aspects  of  task  performance  that  may  be  interesting  from  the 
interviewer’s  point  of  view  but  that  could  be  glossed  over  or  omitted  completely  if 
interviewees  were  left  to  their  own  devices.  This  feature  helps  combat  the  well-known 
knowledge  elicitation  problem  of  experts  being  unable  to  verbalize  their  expertise. 

•  Since  data  are  collected  in  the  context  of  prefabricated  scenarios,  the  variety  of  data  is 
limited  by  the  creativity  of  those  designing  the  scenarios.  This  “limitation”  can  provide 
an  advantage.  Careful  scenario  design  (incorporating  input  from  SMEs)  can  produce  a 
wide  variety  of  challenging  and  realistic  events.  Further,  scenario  design  allows 
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researchers  to  play  an  active  role  in  guiding  the  data  collection.  For  example,  being 
interested  in  teamwork,  researchers  could  design  scenarios  for  which  team  interactions 
are  a  critical  part  of  mission  execution.  Other  methods  of  elicitation  depend  on  the 
interviewee’s  prior  experience  (e.g.  the  critical  incident  method),  which  may  or  may 
not  include  target  concepts  (e  g.  teamwork)  as  important  factors. 

•  The  use  of  a  limited  number  of  standard  scenarios  allows  for  direct  comparison  of  data 
across  subjects:  for  example,  one  could  compare  data  from  experts  and  novices 
confronted  with  the  same  scenario. 

•  Since  interviewees  are  called  upon  to  make  decisions  in  real  time  and  to  discuss  them 
soon  after  making  them,  data  are  likely  to  reflect  the  true  motivations,  cues  and 
strategies  actually  used  in  making  decisions,  whereas  these  factors  may  become 
distorted  when  eliciting  data  retrospectively. 

Such  features  foster  the  collection  of  comprehensive,  detailed,  and  accurate  data. 
The  relative  strengths  of  performance  interviews  and  those  of  retrospective  techniques 
such  as  critical  incident  interviews  have  not  been  empirically  compared,  and  it  is  beyond 
the  scope  of  this  report  to  make  such  a  comparison.  One  way  of  viewing  the  two  methods 
is  that  critical  incident  interviews  target  experience,  whereas  performance  interviews 
target  experience  in  practice.  According  to  this  view,  the  interview  methods  are 
complementary  to  one  another.  In  this  report  we  augment  general  interviews  with  both 
critical  incident  interviews  and  a  variant  of  performance  interviews,  which  we  call  “quasi¬ 
performance”  interviews.  The  procedures  we  used  for  each  interview  method  are 
described  in  detail  in  the  next  section.^ 


^  Our  original  intention  was  to  interview  WDs  immediately  after  they  had  finished  a  simulated  mission  on 
the  C^-STARS  high-fidelity  AW  ACS  simulator  at  Brooks  Ara,  dissecting  with  them  a  videotape  of  their 
own  recent  performance.  This  method  resembled  a  performance  interview  as  closely  as  possible  given  the 
constraints  imposed  by  studying  such  a  dynamic  task.  Before  doing  so,  we  decided  to  make  an  exploratory 
trip  to  Tinker  AFB  in  order  to  a)  conduct  general  interviews  with  the  WDs  stationed  there,  and  b)  collect 
pilot  data  using  the  videotape  review  method  just  described.  Since  we  did  not  have  access  to  a  simulator  at 
Tinker,  we  used  existing  videotapes  of  a  team  conducting  a  mission  on  C’-STARS  and  asked  interviewees 
to  answer  questions  “as  if’  they  were  one  of  the  WDs  on  the  videotape,  otherwise  proceeding  with  the 
interviews  largely  as  plaimed.  We  felt  that  these  initial  interviews  would  yield  useful  information,  but 
more  importantly  would  serve  to  hone  our  interview  structure  and  method,  tailoring  it  to  the  structure  of 
the  WD  task  for  use  in  the  “real”  data  collection  using  C^-STARS.  Unfortunately,  we  were  never  able  to 
access  WDs  as  originally  planned,  and  so  were  not  able  to  conduct  real  performance  interviews.  The 
“quasi-performance”  interview  data  used  in  the  current  study  are  in  fact  the  pilot  data  collected  on  that 
initial  trip  to  Tinker.  Further,  there  are  a  number  of  drawbacks  to  using  this  data,  the  main  ones  being  a) 
we  did  not  interview  as  large  a  number  of  experienced  WDs  as  we  had  hoped  to,  b)  the  “lane  defense” 
configuration  depicted  on  the  videotape  does  not  usually  generate  much  team  interaction,  c)  the  interview 
structure,  based  loosely  on  PARI,  was  extremely  provisional  and  was  not  a  good  fit  to  the  structure  of  the 
WD  task,  and  d)  the  technique  was  not  a  true  performance  interview  (since  interviewees  were  reviewing 
someone  else’s  performance,  they  were  not  put  in  the  position  of  actually  making  decisions,  and  so  we 
would  expect  that  data  about  the  cognitive  underpinnings  of  task  performance  would  be  less  accurate).  In 
this  study  we  have  called  these  interviews  “quasi-performance”  interviews,  in  that  they  exploit  some,  but 
not  all,  of  the  strengths  of  performance  interviews  but  not  all  of  them.  Nevertheless,  although  the  use  of 
these  data  restricted  the  scope  of  our  analysis,  the  description  we  were  able  to  derive  is  no  less  valuable. 
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METHODOLOGY 


There  are  a  large  number  of  knowledge  elicitation  techniques  for  describing  and 
documenting  the  knowledge  underlying  task  performance  (see  Cooke,  1994  for  a  review). 
As  mentioned  before,  we  used  a  mixture  of  general  interviews,  critical  incident  interviews, 
and  a  “quasi-performance”  to  target  knowledge  at  different  levels  of  detail  and  to 
converge  on  a  coherent  set  of  findings  useful  for  the  design  of  SynTEAM.  Here  we 
describe  the  methods  used  in  detail. 


Preparation 

— General  Interviews 

General  interviews  were  used  to  gather  broad  information  about  the  WD  task  and 
to  elicit  context-free,  stable  domain  knowledge.  During  the  initial  stages  of  this  project  we 
analyzed  WD  documentation  and  watched  videotapes  of  teams  of  WDs  performing 
missions  on  C^-STARS,  a  high-fidelity  AW  ACS  simulator.  We  also  had  the  opportunity  to 
talk  with  knowledgeable  SMEs  on  a  number  of  issues  and  questions  that  arose  during 
this  time.  Based  on  our  understanding  of  the  WD  task  as  constructed  from  these  activities, 
we  compiled  a  general  interview  guide  (Appendix  A,  Section  A)  consisting  of  both 
questions  that  we  wanted  answered  in  a  fairly  direct  manner  and  questions  about  issues 
with  which  we  were  already  familiar,  but  on  which  we  wanted  another  perspective. 
Questions  were  designed  to  provide  general  background  on  what  is  involved  in  being  a 
WD,  task  characteristics,  task  demands,  typical  operating  characteristics,  team 
interactions,  and  challenges  facing  WDs.  A  number  of  the  questions  in  the  general 
interview  guide  were  taken  from  the  “Knowledge  Audit”  inventory  described  by  Pliske  et 
al.  (1997). 

—Critical  Incident  Interviews 

Critical  incident  interviews  were  used  to  sample  the  decisions  that  WDs  make  in 
stressful  situations,  and  to  examine  the  characteristics  of  these  decisions.  Our  critical 
incident  interview  guide  (Appendix  A,  Section  B)  was  fashioned  after  that  used  by  Klinger 
et  al.  (1993),  but  was  aimed  explicitly  at  eliciting  and  exploring  incidents  in  response  to 
which  teamwork  was  a  major  factor. 

— Quasi-Performance  Interviews 

The  quasi-performance  interviews  were  intended  to  provide  detailed  information 
about  task  performance  and  its  underlying  knowledge.  Preparation  for  these  interviews 
consisted  of  selecting  and  parsing  a  videotape  to  serve  as  a  stimulus,  and  compiling  an 
interview  guide  of  questions  to  ask.  First,  we  selected  a  videotape  of  a  WD  team 
performing  a  simulated  mission  on  C^-STARS,  the  high  fidelity  AW  ACS  simulator  located 
at  Brooks  AFB.  The  simulator  has  three  WD  consoles  arranged  in  a  row  with  an  SD 
console  located  behind  them  (just  like  on  the  AW ACS  aircraft).  The  videotape  depicted 
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the  contents  of  the  center  WD’s  scope  only.  All  WDs’  voices  were  clearly  audible  on  the 
video  tape.  Pilots’  voices  could  also  be  heard,  albeit  faintly:  hence,  interviewees  could  hear 
everything  that  was  going  on,  but  could  only  see  the  scope  of  the  center  WD.  The  picture 
on  the  videotape  mirrored  the  center  WD’s  scope  exactly,  that  is,  if  the  center  WD 
zoomed  in  or  out  on  his  scope  then  the  videotape  reflected  this. 

The  mission  captured  on  the  tape  was  called  Aaragon,  a  mission  in  which  the  WDs 
had  to  control  fighters  defending  a  fictional  country  against  attacks  by  a  fictional  enemy, 
and  also  clear  the  way  for  fnendly  strike  aircraft  to  penetrate  enemy  airspace. 
Responsibilities  were  divided  among  the  WDs  geographically  (  lane  defense  ).  the  AOR 
was  divided  by  straight  lines  into  three  horizontal  lanes  (North,  Center,  and  South),  with 
the  center  WD  controlling  the  center  lane.  This  particular  tape  was  selected^because 
Aaragon  was  identified  by  a  SME  familiar  with  the  studies  carried  out  on  C^-STARS  as 
the  most  difficult  of  the  simulator  missions  for  which  videotapes  were  available,  and  the 
team  on  the  tape  was  identified  by  the  SME  as  the  best  performing  team.  Further,  while 
the  lane  defense  configuration  is  not  particularly  conducive  to  team  interactions  (and  is  not 
commonly  used  now),  nearly  all  team  interactions  on  the  tape  involved  the  center  WD, 
because  his  lane  is  adjacent  to  both  the  North  and  South  lanes. 

We  reviewed  the  videotape  with  the  aid  of  a  SME,  and  identified  two  encapsulated 
“scenarios”  for  use  as  foci  in  the  quasi-performance  interviews.  These  scenarios  fit  our 
criteria  of  including  both  intense  action  and  team  interaction.  The  first  scenario,  the  air 
refueling”  scenario,  was  an  eleven-minute  section  of  tape  in  which  the  center  WD  was 
coordinating  and  controlling  a  number  of  air  refuelings  involving  his  and  others’  aircraft 
while  under  time  pressure.  The  “air  battle”  scenario  was  a  twenty-minute  tape  section 
during  which  an  air  battle  took  place  on  the  boundary  between  the  Center  and  the  North 
lanes.  In  order  that  discussions  with  interviewees  be  manageable,  we  asked  a  SME  to 
divide  each  scenario  into  segments,  one  to  three  minutes  in  length,  containing  sets  of 
events  that  were  both  cohesive  enough  to  be  interpretable  by  interviewees  as  a  whole  , 
and  significant  enough  to  warrant  discussion.  These  criteria  were  used  to  identify  nine  tape 
segments  for  the  air  refueling  scenario  and  eleven  segments  for  the  air  battle  scenario. 

Finally,  we  were  aware  that  simulator  missions  typically  build  up  in  intensity  over 
time,  allowing  WDs  to  “ease  in”  to  the  high  workload  encountered  in  the  middle  of  such 
missions.  Both  of  our  scenarios  were  taken  from  the  middle  of  the  tape,  well  into  the 
overall  mission.  Presenting  the  scenarios  “cold”  would  have  deprived  interviewees  of  the 
context  needed  to  reasonably  interpret  what  is  going  on.  Hov/ever,  playing  the  enure  tape 
up  to  the  start  of  each  scenario  would  take  too  much  time.  To  provide  sufficient  context  m 
a  reasonable  amount  of  time,  a  SME  identified  “context  building”  segments  to  show  the 
interviewees  prior  to  showing  each  scenario.  The  first  six  minutes  of  the  tape  was 
identified  as  a  “general  introduction”  to  familiarize  interviewees  with  the  mission  set-up 
and  the  taped  WDs’  voices.  We  then  identified  “scenario  introduction”  segments 
immediately  prior  to  each  scenario.  These  were  intended  to  provide  interviewees  with 
enough  immediate  context  to  get  them  up  to  speed  for  the  following  scenario.  The 
introductions  lasted  nine  minutes  for  the  air  refueling  scenario  and  just  under  three  minutes 
for  the  air  battle  scenario.  The  time  difference  between  the  scenario  introduction  segments 
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was  due  to  there  being  a  higher  density  of  events  just  prior  to  the  air  battle  scenario  than 
there  was  just  prior  to  the  air  refueling  scenario. 

The  quasi-performance  interview  guide  (Appendix  A,  Section  C)  was  built  around 
a  set  of  situation  awareness  probes  designed  to  assess  each  interviewee’s  understanding  of 
the  situation  at  the  beginning  and  end  of  each  scenario,  and  a  set  of  questions  based  on 
those  used  in  PARI  interviews  (Hall  et  al.,  1995)  designed  to  elicit  the  knowledge  and 
procedures  underlying  task  performance.  In  PARI  interviews,  interviewees  step  through  a 
realistic  problem,  at  each  step  answering  set  questions  aimed  at  eliciting  Precursors 
(reasons  for  actions).  Actions  (what  the  inter\dewee  would  do).  Results  (of  the  actions), 
and  Interpretations  (of  the  results),  hence  the  “PARI”  acronym.  Treating  the  short 
videotape  segments  as  “steps”  analogous  to  PARI  solution  steps,  we  oriented  our 
questions  toward  the  WD  task  as  we  understood  it,  while  maintaining  the  basic  PARI 
categories  as  best  as  we  could. 

During  the  quasi-performance  interviews,  since  interviewees  were  watching  other 
WDs  on  a  videotape  rather  than  performing  the  task  themselves,  we  could  not  have 
interviewees  generate  actions  and  then  follow  through  on  the  results  and  interpretations  of 
those  actions,  as  in  a  PARI  interview.  To  address  this  problem,  we  generated  two  separate 
sets  of  probes,  the  “next  action”  probes  and  the  “last  action”  probes.  Immediately  before 
each  tape  segment  was  played,  the  interviewee  was  asked  to  imagine  that  he  was  the  WD 
on  the  videotape  (the  center  WD),  and  the  “next  action”  probes  were  used  to  determine 
what  the  interviewee  would  do  and  the  reasoning  behind  these  actions  (just  as  is  done  in 
PARI).  Immediately  following  each  tape  segment,  the  “last  action”  probes  were  used  to 
identify  what  actually  happened  during  the  segment,  to  interpret  these  events,  and  to  elicit 
a  critique  of  the  performance  of  the  WD  on  the  videotape. 


Procedures 

— General  and  Critical  Incident  Interviews 

Researchers  interviewed  WDs  according  to  the  interview  guides.  For  the  most 
part,  the  interview  guides  were  not  meant  to  be  used  verbatim,  and  researchers  remained 
flexible  and  opportunistic  throughout  each  interview.  Although  the  general  and  critical 
incident  interview  formats  were  designed  independently  of  one  another,  it  proved  more 
efficient  to  embed  the  critical  incident  interviews  within  the  general  interviews,  since  WDs 
would  often  start  describing  critical  incidents  in  response  to  general  interview  questions. 
When  this  happened,  we  seized  the  opportunity  to  explore  the  critical  incidents  in  depth  as 
they  occurred  spontaneously,  rather  than  trying  to  elicit  critical  incidents  “cold”  in 
separate  interviews.  General/critical  incident  interviews  took  from  two  to  four  hours  to 
complete,  usually  filling  the  entire  time  for  which  each  WD  was  available  to  us. 
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— Quasi-Performance  Interviews 

WDs  were  interviewed  in  groups  of  up  to  3.  After  an  initial  introduction  to  the 
general  aims  of  the  study,  a  verbal  description  of  the  contents  of  the  videotape,  and  a 
description  of  how  the  interview  would  proceed,  interviewees  were  given  mission  briefing 
packets  to  study.  These  packets  were  identical  to  those  provided  to  the  WDs  on  the 
videotape  prior  to  their  performing  the  simulated  mission.  This  study  period  generally  took 
around  15  minutes,  after  which  WDs  were  given  the  opportunity  to  ask  the  researchers 
questions.  One  of  the  interviewers  then  reiterated  the  interview  procedure,  stating  that 
interviewees  would  be  watching  a  videotape  of  the  center  WD’s  scope  taken  during  a 
simulated  mission  employing  the  lane  defense  configuration,  and  that  interviewees  were  to 
imagine  themselves  as  the  center  WD.  While  the  “general  introduction”  portion  of  the 
videotape  was  played  on  a  large-screen  TV,  one  of  the  interviewers  provided  commentary, 
orienting  the  interviewees  to  features  on  the  screen  and  identifying  WDs’  voices.  At  the 
end  of  the  general  introduction,  the  same  interviewer  fast-forwarded  the  videotape  such 
that  the  picture  remained  visible  on  the  screen,  albeit  at  high  speed  and  without  the 
accompanying  audio  track.  This  procedure  allowed  interviewees  to  follow  the  unfolding  of 
the  scenario,  providing  additional  context,  while  also  saving  time  compared  to  playing  the 
videotape  at  normal  speed.  The  interviewer  again  provided  commentary  on  the  events  on 
the  screen  during  this  time.  The  interviewer  stopped  providing  commentary  when  the  tape 
reached  the  beginning  of  the  introduction  for  the  chosen  scenario,  returned  the  tape  to 
normal  speed,  and  left  it  to  play  up  until  the  beginning  of  the  actual  scenario. 

Each  interviewee  then  paired  up  with  one  of  the  interviewers  in  front  of  a  20” 
television  and  VCR.  Interviewees  were  given  paper  copies  of  the  terrain  map  depicted  on 
the  videotape,  and  were  asked  to  mark  the  positions,  headings,  and  call-signs  of  both 
friendly  and  enemy  aircraft,  as  they  remembered  them  to  be  just  before  the  tape  was 
turned  off.  Interviewees  were  also  questioned  about  the  current  state  of  the  scenario, 
using  the  “situation  awareness”  probes  (Appendix  A,  Section  C).  The  purpose  of  this 
procedure  was  to  assess  each  interviewee’s  “situation  awareness”;  that  is,  whether  or  not 
the  interviewee  accurately  understood  what  was  going  on.  This  was  done  so  that 
interviewees’  comments  could  later  be  interpreted  in  light  of  how  well  they  understood  the 
situation.  Follo\ving  the  situation  awareness  probes,  the  interviewer  started  stepping 
through  the  videotape,  asking  appropriate  questions  at  appropriate  points:  before  each 
segment  of  tape  was  played  the  interviewer  asked  the  “next  action”  probes  (Appendix  A 
Section  C),  and  after  each  segment  of  tape  the  interviewer  asked  the  “last  action”  probes 
(Appendix  A  Section  C).  Interviewers  were  encouraged  to  be  flexible  and  opportunistic  in 
their  questioning.  This  procedure  was  repeated  until  the  end  of  the  scenario  had  been 
reached,  after  which  interviewees’  situation  awareness  was  again  assessed  using  the  same 
method. 


Participants 

Twenty  male  AW  ACS  Weapons  Directors  (WDs)  served  as  interviewees.  Nineteen 
of  these  were  currently  stationed  at  Tinker  AFB,  OK,  while  one  was  stationed  at  Brooks 


12 


AFB,  TX.  Most  interviewees  participated  in  either  a  general  interview  and  a  critical 
incident  interview  or  in  a  quasi-performance  interview  using  one  of  the  two  scenarios. 
However,  where  time  allowed,  some  interviewees  covered  more  than  one  scenario  in  their 
quasi-performance  interview,  while  one  interviewee  participated  in  a  general/critical 
incident  interview  and  covered  both  scenarios  in  the  quasi-performance  interview.  The 
experience  and  participation  level  of  each  WD  is  documented  in  Table  I. 

Table  1.  Experience  and  Participation  Level  of  Participating  WDs 


WD 

Loc. 

WD 

Exp.* 

Special  Exp. 

General/ 
Crit.  Inc. 

Refueling 

Scenario 

Battle 

Scenario 

1 

Tinker 

2y  3ni 

Instructor 

V 

2 

Tinker 

2y 

V 

3 

Tinker 

lOm 

V 

4 

Tinker 

10m 

v 

5 

Tinker 

4y 

V 

6 

Tinker 

ly  4m 

V 

7 

Tinker 

lOm 

V 

8 

Tinker 

4y 

SD 

V 

9 

Tinker 

4y  6m 

Instmctor 

V 

10 

Tinker 

2y 

v/ 

11 

Tinker 

4y 

SD 

V 

12 

Tinker 

4m 

V 

13 

Tinker 

ly 

V 

14 

Tinker 

SD 

V 

15 

Tinker 

ly 

V 

16 

Tinker 

5m 

V 

17 

Tinker 

V 

18 

Tinker 

2y 

V 

19 

Tinker 

10m 

< 

V 

20 

Brooks 

5y 

Instmctor 

V 

V 

V 

*  First  year  of  experience  is  spent  in  training. 
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To  summarize  the  information  in  Table  I,  eight  WDs  had  under  one  year  of 
experience  (i.e.,  they  were  still  trainees),  six  had  between  one  and  three  years  of 
experience,  and  six  had  over  three  years  of  experience.  Of  this  last  group,  three  were  SDs 
and  three  were  instructors.  Nine  WDs  participated  in  general/critical  incident  interviews 
and  twelve  participated  in  the  quasi-performance  interviews  (with  one  WD  participating  in 
both  formats).  Within  the  quasi-performance  interviews,  nine  covered  the  refueling 
scenario  and  five  covered  the  battle  scenario,  with  two  WDs  covering  both  scenarios. 
Table  II  shows  the  experience  level  of  WDs  broken  out  by  interview  participation. 

Table  II.  Experience  Level  of  WDs  Organized  bv  Interview  Participation 


Interview  Format 

n 

Experience  Level  (y-m) 

Mean  Experience  (y-m) 

General/Critical  Incident 

9 

0-5,  0-10,  1,  1-4,  4,  4,  4-6,  5-0,  7-0 

3-0 

Performance:  Refueling 

9 

04, 0-10,  0-10,  0-10,  1-0,  2-0,  2-3, 
3-0,  5-0 

1-9 

Performance:  Battle 

5 

0-10,  2-0,  2-0,  4-0,  5-0 

2-9 

DATA  ANALYSIS 


All  interviews,  recorded  on  audio  tape,  were  transcribed  immediately  upon 
returning  from  the  data  collection  trip.  Transcripts  were  augmented  by  notes  the 
interviewers  had  made  during  the  course  of  the  interviews.  After  transcription  was 
completed,  it  was  clear  that  the  PARI  structure  we  had  used  for  the  quasi-performance 
interviews  did  not  fit  the  structure  of  the  WD  task.  The  PARI  structure  was  derived  from 
a  model  of  problem  solving  based  on  research  on  the  troubleshooting  methods  of  avionics 
technicians.  While  this  structure  does  generalize  to  tasks  other  than  avionics 
troubleshooting  (eg.,  Fahey  et  al.,  1997),  it  did  not  capture  the  dynamism  of  the  WD  task, 
nor  did  it  well  represent  multitasking.  To  better  understand  the  data  (and  to  formulate  a 
model  that  could  generate  an  interview  structure  for  further  data  collections),  we 
constructed  a  schema  to  describe  the  cyclical  nature  of  the  process  by  which  WDs  interact 
with  their  environment.  We  then  independently  reviewed  our  transcripts  and  identified 
“themes”  for  further  analysis,  using  the  new  schema  as  a  guide  to  interpreting  the  data 
(replacing  the  PARI  schema,  in  the  context  of  which  the  data  was  collected).  Themes  were 
predefined  as  issues  facing  WDs  during  the  course  of  mission  performance  that  cause  them 
particular  difficulties'*.  After  a  number  of  brainstorming  sessions,  in  which  these  themes 


''  Themes  were  derived  from  data  from  all  three  interview  formats.  Although,  on  the  whole,  the  general 
interviews  provided  background  knowledge  while  the  critical  incident  and  quasi-performance  interviews 
provided  detailed,  concrete  examples  of  task  performance,  there  was  nonetheless  some  overlap  between 
formats.  For  example,  in  the  general  interviews  interviewees  often  illustrated  the  points  they  were  trying 
to  make  with  concrete  examples  of  task  performance,  while  in  the  quasi-performance  interviews  specific 
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were  discussed  and  explored  in  more  depth,  we  decided  to  use  “decision  requirements 
tables”  to  represent  our  data,  since  these  are  a  good  way  of  standardizing  verbal  protocol 
data.  In  turn,  the  tables  that  we  each  produced  were  consolidated  and  compiled,  and  a  task 
description  and  specifications  for  SynTEAM  design  were  produced. 

In  the  next  section  of  the  report,  we  describe  the  formulation  of  our  interpretative 
schema  for  the  WD  task,  and  then  describe  the  process  through  which  we  derived  the 
decision  requirements  tables. 


Describing  the  Interaction  Between  WDs  and  their  Environment 

There  were  two  reasons  for  constructing  a  schema  to  describe  the  WD  task.  The 
first  was  to  motivate  an  interview  structure  that  would  better  facilitate  the  collection  of 
data  in  future  interviews  with  WDs,  in  the  same  way  that  PARI  facilitates  collecting  data 
from  troubleshooters.  The  second  was  to  provide  a  framework  for  organizing  the  data  that 
we  had  already  collected  using  the  PARI  framework. 

— Underlying  Model 

The  main  problem  with  using  PARI  for  collecting  data  from  WDs  was  that  the 
PARI  structure  was  a  poor  fit  to  the  WD  task.  The  WD  task  is  very  dynamic  and  event- 
driven,  whereas  PARI  is  most  useful  for  capturing  performance  on  tasks  driven  by  stable 
goals  such  as  troubleshooting  tasks.  This  suggested  that  another  structure  would  be  more 
appropriate  for  describing  the  WD  task,  so  we  set  about  developing  a  schema  that  would 
more  accurately  capture  the  thought  processes  underlying  the  WD  task.  The  initial  step  in 
developing  the  new  schema  was  to  identify  existing  generic  models  of  cognitive  processing 
that  could  motivate  the  new  structure,  in  the  same  way  that  a  model  of  problem  solving  as 
serial  hypothesis  testing  had  motivated  PARI.  We  reviewed  a  number  of  models  that 
address  real-time,  dynamic  tasks:  Observe-Orient-Decide-Act  (ODD A)  Loops  (Whitaker 
&  Kuperman,  1996),  an  enhanced  version  of  Neisser’s  (1976)  Perception-Action  Cycle 
(Adams,  Termey,  &  Pew,  1995),  and  a  model  of  human-computer  interaction  described  by 
Norman  (1984).  All  models  were  very  similar  in  that  they  described  a  cyclical  process  of 
perception,  thought,  and  action:  for  our  purposes,  the  crucial  step  was  how  to  carve  the 
cycle  into  appropriate  segments  in  order  to  provide  a  schema  to  describe  the  specific  task 
at  hand.  We  first  identified  the  informational  and  cognitive  categories  important  to  task 
performance,  then  arranged  these  into  a  perception/action  cycle  that  seemed  to  well 
describe  both  how  WDs  carry  out  the  many  tasks  they  are  called  upon  to  perform  and  how 
they  integrate  these  tasks  into  the  job  as  a  whole.  Finally,  we  verified  that  this  schema  was 
a  good  fit  to  the  WD  task  as  a  whole  by  retrofitting  it  to  the  quasi-performance  interview 
data  we  had  already  collected. 

— An  Initial  Schema  to  Describe  the  WD  Task 


actions  were  often  related  to  general  principles.  All  analyses  presented  in  this  report  reflect  data  from  all 
interview  formats. 
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The  initial  structure,  called  PIGPEN,  derives  from  general  frameworks  (listed  in 
the  previous  section)  as  applied  WD  performance  (as  determined  from  the  interview  data): 

P  -  Perceive  events  in  the  environment 
I  -  Interpret  these  events 
G  -  Attach  high-level  Goals  to  these  events 

P  -  Integrate  these  events  into  a  “big  picture”  and  formulate  a  Plan  (or  Plans) 

E  -  Evaluate  this  plan  (these  plans) 

N  -  Decide  on  a  specific  set  of  Next  Actions  to  execute  the  chosen  plan 

First,  the  WD  perceives  events  in  the  environment.  For  example,  the  WD  might 
notice  a  new  track  in  hostile  territory,  a  radio  call  that  a  pilot  bailed  out,  a  “Bingo”  call 
from  one  of  his  fighters,  and  an  undirected  comment  made  by  the  WD  controlling  the 
North  lane  that  there  are  more  hostile  tracks  in  his  lane  than  his  fighters  can  handle.  He 
then  interprets  these  events.  For  example,  the  new  track  may  be  inferred  to  be  new  hostile 
aircraft  coming  into  the  lane,  the  bail-out  might  be  identified  as  a  particular  aircraft  being 
shot  down,  the  bingo  call  would  be  interpreted  as  an  aircraft  needing  gas,  and  the 
comment  made  by  the  other  WD  might  be  interpreted  as  indicating  that  the  other  WD  is 
overloaded  and  needs  help.  Next,  the  WD  attaches  goals  to  each  of  these  events.  For 
example,  the  new  track  would  have  to  be  intercepted,  the  downed  pilots  rescued,  the  low- 
fuel  aircraft  refueled,  and  the  north  WD  helped  if  possible.  After  all  the  events  of  interest 
are  noted  and  understood,  the  WD  forms  a  game  plan  (or  plans)  for  attaining  the  goals. 
This  is  where  WDs  allocate  their  resources  to  targets,  and  plan  what  to  do  next.  The  WD 
then  evaluates  the  plan  or  plans  and,  once  decided  on  a  particular  course  of  action, 
identifies  individual  actions  in  the  order  that  they  would  be  performed. 

— Testing  the  Schema 

We  tested  the  schema  by  retrofitting  it  to  the  existing  quasi-performance  interview 
data,  step  by  step.  Although  we  allowed  that  the  interview  data  might  not  have  sufficient 
detail  to  allow  us  to  test  the  utility  of  some  aspects  of  the  new  schema,  we  nevertheless 
found  several  shortcomings.  First,  in  practice  it  is  difiBcult  to  distinguish  between  the 
“perceive”  and  “interpret”  categories:  for  example,  the  WD  might  say  “there’s  an 
unknown  track  over  hostile  territory:  it  is  probably  a  bogey”.  It  is  not  natural  for  WDs  (or 
anyone  else  for  that  matter)  to  identify  something  without  attaching  meaning  to  it.  Second, 
WDs  do  not  seem  to  decide  on  goals  for  dealing  with  events,  except  at  the  most  abstract 
levels,  before  they  form  an  explicit  plan.  Although  events  are  often  unambiguously  paired 
with  goals  at  an  abstract  level  (e  g.,  the  ROE  might  dictate  that  an  incoming  hostile 
aircraft  has  to  be  intercepted)  the  method  by  which  each  goal  is  achieved  cannot  be 
decided  out  of  the  context  of  all  else  that  is  going  on.  For  example,  although  a  hostile 
aircraft  needs  to  be  intercepted,  exactly  how  to  do  this  will  depend  on  how  the  WD’s 
limited  resources  are  otherwise  committed:  the  closest  set  of  fighters  might  not  be  able  to 
make  the  intercept  without  endangering  some  other  part  of  the  mission.  After  assessing 
the  situation  (integrating  the  new  events  into  all  else  that  is  going  on),  WDs  typically 
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formulate  and  evaluate  one  or  more  plans.  Strategies,  prior  experience  (“have  I  seen 
something  like  this  before?”),  and  the  priority  level  of  each  goal  are  all  factors  in 
formulating  a  plan.  For  this  reason,  the  “Goal”  step  should  be  absorbed  into  the  “Plan” 
step,  and  “Priorities”  should  be  assigned  to  each  event  after  the  planning  stage  in  order  to 
detail  the  exact  actions  to  perform  next.  In  reality,  the  planning  stage  is  a  major  part  of  the 
WD’s  job.  Because  in  a  typical  mission  there  are  multiple  constraints  to  satisfy  at  any  one 
time  and  limited  resources  with  which  to  do  so,  WDs  often  have  to  make  tradeoffs  in 
order  to  satisfy  the  maximum  number  of  goals  to  the  maximal  extent.  This  involves  some 
trial  and  error,  as  plans  are  formulated  and  evaluated.  In  this  sense,  the  WD  task  is  unlike 
many  other  “naturalistic  decision  making”  situations  in  which  the  accuracy  of  the  situation 
assessment  is  the  prime  determinant  of  whether  a  good  decision  is  made  (Klein,  1993). 


— The  Final  Schema 

Based  on  our  limited  data,  we  decided  on  a  final  schema  (Figure  3).  This  schema 
was  used  to  interpret  the  data  we  had  already  collected.  Currently,  this  schema  could  be 
used  to  generate  a  new  interview  structure  that  would  correspond  more  closely  to  the  way 
WDs  actually  think  than  did  the  PARI  structure,  and  should  lead  to  the  collection  of  better 
data.  In  the  future,  additional  data  from  true  performance  interviews  could  be  used  to 
expand  the  schema  into  a  more  complete  model  of  WD  performance. 


Evaluate 


I  1 

Action 


WD 


Environment 


Comi^nication 


Figure  3:  A  Schema  for  the  WD  Task 


From  Verbal  Protocols  to  Decision  Requirements  Tables 
— Brainstorming  Sessions 

Each  of  us  independently  reviewed  our  interview  transcripts  and  identified  WD 
“themes”,  defined  as  issues  facing  WDs  that  cause  particular  difficulties  or  aspects  of  the 
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task  that  are  particularly  challenging,  using  the  schema  outlined  above  as  a  guide.  These 
themes  formed  the  foci  of  our  discussions  in  subsequent  meetings.  In  these  meetings,  it 
became  clear  that  the  themes  could  be  organized  according  to  two  broad  categories  that 
seemed  to  capture  what  WDs  spend  most  of  their  time  doing:  gathering  information  from 
the  environment  to  maintain  situation  awareness  and  using  their  information  and 
knowledge  to  make  decisions  that  affect  the  environment.  We  also  considered  a  third 
category,  teamwork,  to  focus  attention  on  our  main  concern  here. 

A  further  brainstorming  session  focused  on  what  we  wanted  to  learn  from  our 
data.  Based  on  our  data’s  limitations,  we  decided  to  list  the  tasks  and  actions  performed 
by  WDs,  and  also  the  cognitively-challenging  events  and  scenarios  faced  by  WDs.  From 
these  lists  we  made  our  decision  requirements  tables. 

— Decision  Requirements  Tables 

As  a  standard  format  for  representing  our  inventories  of  tasks,  actions,  and  events, 
we  chose  a  format  inspired  by  “decision  requirements  tables”,  used  extensively  by  Klein 
and  his  colleagues  as  a  way  to  represent  verbal  interview  data  in  a  useful  format. 

Typically,  decision  requirements  tables  lay  out  information  such  as  a  description  of  a 
decision  made,  the  cognitive  requirements  of  that  decision,  and  the  cues  and  information 
used  in  making  the  decision.  However,  there  is  no  standard  format  for  these  tables,  and  we 
customized  as  we  saw  fit.  Since  the  WD  task  is  so  multifaceted,  with  decisions  being  made 
at  a  number  of  levels,  we  used  three  formats; 

1)  Subtasks  Analysis  Tables 


Subtask 


Decisions 


Information/Cues 


Component  Actions 


The  subtasks  analysis  tables  were  initially  used  as  a  first  pass  at  decomposing  the 
WD  task,  to  enumerate  and  analyze  the  major  subtasks  performed  by  WDs.  SynTEAM 
should  at  least  support  performance  of  these  subtasks.  However,  as  we  got  further  into 
the  data  analysis  we  realized  that  these  tables  were  useful  for  representing  subtasks  right 
down  to  the  switch  action  level. 

2)  Actions  Analysis  Tables 


Action 


Cognitive  Workload 


In  these  tables,  we  estimated,  at  a  qualitative  level,  the  workload  generated  by  each 
action  (task  component)  identified  in  the  subtasks  analysis  tables. 
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3)  Events  Analysis  Tables 

I 

Event/ Action 


Challenges 


Decisions 


Errors 


These  tables  describe  the  enAnronmental  events  typically  encountered  by  WDs,  and 
characterize  the  decisions  made  in  response  to  these  events.  These  tables  thus  capture 
decisions  made  at  the  strategic  and  tactical  planning  levels,  as  opposed  to  the  subtask  level 
decisions  captured  by  the  subtasks  analysis  tables.  For  example,  the  events  analysis  table 
would  look  at  decisions  such  as  how  to  respond  to  a  hostile  threat,  while  the  subtasks 
analysis  table  would  capture  the  decisions  that  need  to  be  made  once  it  has  been  decided 
that  the  appropriate  response  is  to  send  a  particular  fighter,  such  as  the  best  route  to  send 
it  on. 


All  tables  were  pooled  and  duplicates  eliminated.  These  tables  were  then  used  as 
data  to  derive  the  task  description,  presented  in  the  next  section. 

WD  TASK  DESCRIPTION 


The  aim  of  building  SynTEAM  is  to  provide  a  research  platform  on  which  to  study 
the  effects  of  various  interventions  on  the  effectiveness  of  WD-like  teams.  The  challenge 
for  designing  SynTEAM  is  deciding  how  the  real  world  WD  task  can  be  distilled  in  a 
leamable  synthetic  task  while  preserving  the  ecological  validity  of  the  real  task. 
Accomplishing  this  requires  first  identifying  the  task  characteristics  that  define  the  WD 
task  and  drive  WD  team  performance,  then  building  SynTEAM  to  share  these 
characteristics.  To  do  this,  we  generated  a  task  description  from  the  information  captured 
in  the  decision  requirements  tables,  constantly  referring  back  to  the  raw  protocols  for 
validation  and  enhancement.  We  also  derived,  from  the  description,  a  preliminary  set  of 
specifications  for  SynTEAM  design:  these  will  be  presented  following  the  task  description. 

Our  task  description  is  divided  into  two  parts: 

1 .  General  characteristics  or  context-free  attributes  of  the  WD  task:  For  example,  WDs 
can  freely  talk  to  each  other  at  any  time  during  operations.  This  suggests  SynTEAM 
WDs  should  be  similarly  enabled.  It  also  suggests  that  modes  of  communication  might 
be  an  interesting  issue  for  study  in  a  SynTEAM  environment. 

2.  Specific  events  that  occur  during  the  course  of  a  mission:  These  are  most  useful  in 
scenario  design  and  include  not  only  the  descriptions  of  the  events,  themselves,  but 
also  descriptions  of  how  WDs  deal  with  the  events. 


ti 
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General  Characteristics 


Ten  broad  characteristics  of  the  WD  domain  stand  out  as  “defining”  features: 

•  Dynamic,  dense  environment;  WDs  have  to  contend  with  a  large  quantity  of  rapidly 
changing  information.  Situations  can  quickly  develop  from  routine  to  crisis,  and  WDs 
often  work  under  tight  time  constraints.  WDs  must  work  to  stay  on  top  of  the 
situation  at  all  times,  since  falling  even  a  little  behind  a  routine  situation  can  turn 
disastrous  as  the  situation  develops  into  something  more  critical. 

•  Perceptually-demanding;  Because  the  WD  environment  is  so  dynamic,  the  WD  task  is 
extremely  perceptually-demanding.  Not  only  do  WDs  have  to  track  resources  that  are 
moving  around  the  airspace,  but  they  must  also  calculate  complicated  “intercept 
geometry”  to  bring  two  or  more  moving  tracks  together  at  a  predetermined  point  in 
space.  Often,  at  least  one  track  is  not  under  the  WD’s  control,  and  so  a  small  error  in 
the  wrong  direction  could  result  in  a  missed  intercept. 

•  Multitasking:  WDs  often  have  to  perform  more  than  one  task  at  one  time:  for  example, 
a  WD  controlling  fighters  might  be  monitoring  a  number  of  hostile  intercepts  all 
happening  at  once.  Many  tasks  require  continual  attention,  and  so  there  is  often 
considerable  temporal  overlap.  WDs  need  to  prioritize  tasks,  estimating  when  each 
needs  to  be  revisited,  and  paying  more  attention  to  critical  tasks  or  tasks  that  are 
nearing  a  critical  point. 

•  Ill-defined  problems:  WDs  are  often  faced  with  problems  and  situations  to  which  there 
is  no  one  correct  response.  For  example,  there  may  be  more  than  one  way  to  direct  a 
fighter  to  deal  with  an  incoming  hostile  threat,  and  WDs  need  to  project  forward  to 
estimate  which  solution  would  have  the  most  advantageous  outcome. 

•  Division  of  labor:  WDs  cope  with  the  amount  of  work  involved  in  controlling  a 
mission  by  dividing  mission  responsibilities  among  themselves.  There  are  two  common 
ways  of  dividing  responsibility:  1)  geographically,  where  the  AOR  is  subdivided  into 
“lanes”,  and  each  WD  is  responsible  for  everything  that  goes  on  in  one  lane,  and  2) 
functionally,  where  the  AOR  is  not  subdivided,  and  each  WD  performs  a  different 
function  in  support  of  the  missions  in  the  AOR.  Currently,  the  functional  division  of 
labor  is  by  far  the  most  common. 

•  Hierarchical  team  organization;  The  SD  is  a  big  part  of  the  team,  since  a  mission  will 
involve  many  decisions  that  WDs  are  not  authorized  to  make.  How  much  authority 
WDs  have  will  vary  from  mission  to  mission  and  from  SD  to  SD,  but  WDs  do  not 
usually  decide  when  to  scramble  new  aircraft  or  make  large  strategic  decisions  such  as 
which  aircraft  to  commit  to  which  hostile  tracks.  However,  since  WDs  are  in  a 
privileged  position  to  understand  what  is  going  on,  a  WD  will  typically  approach  the 
SD  with  both  the  problem  and  the  solution,  and  the  SD  will  often  agree. 

•  Free  flow  of  information:  The  most  common  way  for  WDs  to  communicate  among 
themselves  is  verbally;  either  over  Net  2  (the  “WD  Channel”)  or  using  “Net  4”  (face  to 
face  communication).  Both  of  these  are  “public”  channels,  in  that  all  WDs  can  hear 
what  is  being  said,  and  so  most  information  being  passed  is  in  principle  accessible  to 


20 


all  WDs.  Further,  since  WDs  can  listen  to  all  the  radio  frequencies,  information  passed 
between  WDs  and  pilots  is  also  effectively  public. 

•  Overlapping  mental  models:  Although  WDs  take  on  different  responsibilities  within  a 
mission,  all  Avill  have  received  the  same  training,  and  so  there  is  no  specialization  of 
function.  For  example,  a  WD  might  control  fighters  in  one  mission,  and  tankers  in  the 
next.  This  means  that  WDs’  mental  models  of  the  WD  task  (stable  task  understanding) 
overlap  considerably.  Each  WD  understands  not  only  his  own  responsibilities  but  also 
the  responsibilities  of  the  other  WDs  on  the  team. 

•  Overlapping  situation  models:  WDs  tend  to  share  a  lot  of  information  vdth  each  other, 
and,  since  they  are  usually  working  within  a  common  airspace  using  public  channels,  it 
is  not  uncommon  for  all  WDs  to  be  highly  aware  of  everything  that  is  going  on. 
However,  during  periods  of  intense  activity  WDs  get  so  involved  in  their  own 
responsibilities  that  they  cannot  keep  up  with  everything  else  going  on. 

•  Opportunistic  cooperation:  The  most  interesting  team  interactions  arise 
opportunistically  during  times  of  crisis  or  high  workload.  Even  though  all  WDs  have 
their  own  responsibilities  and  duties,  they  have  to  be  extremely  flexible,  working 
together  and  taking  on  new  responsibilities  in  order  to  solve  unexpected  problems. 
WDs  often  actively  enlist  the  help  of  other,  less  busy  WDs,  and  they  try  to  look  out  for 
one  another,  helping  out  when  the  need  arises. 

These  broad  characteristics  have  many  ramifications  for  how  WDs  perform  their 

duties.  Next,  we  describe  the  general  characteristics  of  the  WD  task  in  more  detail, 

looking  at  task  demands  and  WD  strategies.  We  have  divided  the  description  into  the 

following  categories: 

•  Team  composition  and  configuration 

•  Mission  characteristics 

•  Task  performance 

•  Information  flow 

•  Teamwork 

•  Interface  issues 

— Team  Composition  and  Configuration 

•  WD  teams  are  usually  comprised  of  at  least  3  WDs  and  an  SD.  All  WDs  receive  the 
same  training,  while  SDs  receive  special  SD  training  in  addition  to  the  training  they  get 
as  a  WD.  WDs  on  a  single  team  often  differ  in  experience  level,  and  team  personnel  is 
often  changed.  The  SD  is  often  not  the  most  experienced  person  on  a  team:  while 
experienced  officers  are  often  promoted  to  SD,  enlisted  personnel  cannot  be  SDs,  and 
so  it  is  not  uncommon  for  SDs  to  have  under  their  command  much  more  experienced 
enlisted  personnel. 
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•  Mission  responsibilities  are  divided  up  among  WDs  depending  on  the  demands  of  the 
particular  mission.  Teams  are  configured  in  anticipation  of  the  events  and  workload 
that  a  particular  mission  will  entail.  The  goal  is  not  to  overload  any  of  the  WDs.  There 
are  two  common  divisions  of  labor,  geographical  and  functional,  with  the  functional 
set-up  being  most  common.  A  common  division  of  labor  is. 

-  Check-in:  when  a  fnendly  aircraft  takes  off,  the  check-in  WD  contacts  it  and 
determines  if  it  is  a  scheduled  flight  according  to  the  Air  Tasking  Order  (ATO).  If  it  is 
scheduled,  the  check-in  WD  determines  whether  it  is  configured  as  described  in  the 
ATO  (“as  ffag’d”*).  If  it  is  not  scheduled,  its  identity,  description  and  mission  are 
determined.  As  aircraft  near  the  AOR,  the  check-in  WD  hands  control  over  to  the  AOR 
WD,  “pushing”  aircraft  onto  the  AOR  WD’s  primary  radio  frequency,  and  informing 
the  AOR  WD  of  any  deviations  from  the  ATO.  It  is  also  the  check-in  WD’s 
responsibility  to  inform  the  AOR  WD  and  the  SD  when  a  scheduled  aircraft  fails  to 
appear.  The  check-in  WD  also  checks  aircraft  back  out  as  they  leave  the  AOR. 

-  AOR  (also  known  as  DCA  [defensive  counter-air]):  the  AOR  WD  controls 
aircraft  within  the  AOR,  coordinating  fighters  to  man  the  Combat  Air  Patrol  (CAP) 
points,  and  overseeing  any  other  activities  that  may  be  going  on  in  the  AOR  (such  as 
escorting  a  diplomatic  flight,  or  controlling  a  rescue  operation).  The  AOR  WD  is 
responsible  for  monitoring  and  intercepting  any  threats  that  materialize  in  the  AOR. 

-  Tanker/HVA  (High  Value  Assets):  as  aircraft  run  out  of  fuel,  the  AOR  WD 
sends  them  to  the  airborne  tankers,  passing  control  over  to  the  Tanker/HVA  WD. 
Refuelings  are  usually  scheduled  during  routine  missions,  but  during  combat  situations 
refuelings  occur  on  an  as-needed  basis.  The  Tanker/HVA  WD  is  then  responsible  for 
on-the-fly  scheduling,  expediting  the  refueling  of  critical  aircraft,  and  returning  to  base 
(RTB)  aircraft  that  cannot  be  accommodated.  After  each  aircraft  is  refueled,  it  is  sent 
back  to  the  AOR,  and  control  is  returned  to  the  AOR  WD.  The  Tanker/HVA  WD  is 
also  responsible  for  controlling  the  high  value  assets  involved  in  a  mission,  such  as 
reconnaissance  aircraft  and  the  AW  ACS  aircraft  itself  In  large  missions,  responsibility 
for  tankers  and  HVAs  might  be  divided  between  two  WDs. 

Other  common  responsibilities  are  “Strike”,  who  controls  and  monitors  strike 
packages  going  in  to  enemy  territory,  and  “Assist”,  who  aids  a  designated  WD  (e.g.. 
Check-in  Assist,  Strike  Assist)  when  workload  is  anticipated  to  be  high. 

The  WD  console  supports  flexible  division  of  responsibility  by  allowing  all  WDs 
access  to  all  system  information,  all  the  while  allowing  WDs  to  customize  the  settings 
on  their  individual  consoles  in  order  to  focus  on  the  most  relevant  information.  An  SD 
oversees  and  directs  the  WDs,  and  is  the  link  between  the  WDs  and  the  MC  and 
between  WDs  and  Ground  Control.  The  SD  is  involved  in  deciding  when  to  scramble 


^  The  ATO  is  made  up  of  smaller  “fragmentary  orders”  for  single  units  such  as  a  flight  of  aircraft.  Hence 
the  term  “as  frag’d”  to  signify  agreement  with  the  fragmentary  order  on  the  ATO. 
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new  aircraft,  and  is  the  WDs’  “authority”  for  making  strategic  decisions  such  as  which 
aircraft  to  commit  to  a  hostile  track  or  whether  or  not  to  shoot  at  a  hostile  track. 

—Mission  Characteristics 

•  Missions  usually  involve  a  number  of  separate  activities,  all  of  which  WDs  must 
monitor  and  control.  The  most  common  activities  are  patrol,  defense,  strike,  escort, 
and  search  and  rescue.  A  single  mission  usually  involves  a  mixture  of  these  activities, 
often  with  a  number  of  activities  going  on  at  once. 

•  Because  responsibilities  are  divided  among  WDs,  different  WDs  become  busy  at 
different  times;  the  “cadence”  of  the  mission  is  such  that  some  WDs  will  be  extremely 
busy  at  the  same  time  that  other  WDs  have  little  to  do.  For  example,  the  check-in  WD 
is  busiest  at  the  beginning  of  the  mission,  while  the  AOR  WD  is  busiest  in  the  middle 
of  the  mission.  It  is  during  such  times  of  differential  workload  that  the  most  team 
interaction  tends  to  take  place,  as  WDs  work  together  to  spread  the  workload  to  avoid 
any  one  WD  becoming  overloaded. 

•  Routine  missions  are  tightly  scheduled.  WDs  know  exactly  who  will  be  flying,  and 
when  and  where  they  will  be  flying.  All  this  information  is  contained  in  the  Air  Tasking 
Order  (ATO).  WDs  have  to  be  vigilant,  noting  deviations  from  the  ATO  and  passing 
this  information  on  to  the  other  WDs  and  the  SD.  However,  when  unexpected  events 
occur,  such  as  an  enemy  attack,  WDs  need  to  improvise  and  the  ATO  becomes  less 
important. 

•  WDs  always  have  to  follow  precise  Rules  of  Engagement  (ROE).  Rules  are  specified 
for  different  levels  of  activity,  from  peacetime  to  war,  with  a  number  of  gradations  in 
between.  The  ROE  tell  the  WD  what  can  be  done  at  each  level,  such  as  how  to 
address  and  intercept  hostile  or  unknown  aircraft.  Before  deciding  upon  an  action, 
WDs  need  to  consider  the  current  ROE  and  make  sure  that  any  decision  is  appropriate 
given  the  current  rules. 

•  Each  mission  is  preceded  by  a  briefing.  During  the  briefing,  WDs  and  the  SD  discuss 
the  mission  plan  and  schedule,  review  the  ROE,  and  work  out  in  advance  how  events 
will  be  dealt  with.  Each  WD’s  roles  and  responsibilities  will  be  clearly  worked  out. 
WDs  make  verbal  “contracts”  with  each  other  regarding  what  and  how  information 
will  be  passed.  WDs  try  to  anticipate  all  that  could  happen  in  a  mission,  and  make 
contracts  establishing  how  they  will  help  each  other  out  in  difficult  situations. 
Contracts  are  a  very  important  part  of  the  total  mission,  and  WDs  take  them  very 
seriously. 

•  The  WDs’  environment  is  very  dynamic,  often  with  many  things  going  on  at  once. 
Even  in  a  routine  mission,  WDs  have  to  prioritize  tasks  efficiently  in  order  to  service 
each  in  a  timely  manner;  attention  must  be  rotated  between  tasks,  and  each  task  must 
be  revisited  at  critical  points.  A  WD  cannot  afford  to  be  lax  even  in  times  of  low 
workload,  because  a  dangerous  situation  can  unfold  very  quickly,  and  if  a  WD  is 
behind  or  unprepared  important  tasks  can  easily  be  forgotten  in  the  context  of  new 
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events.  In  situations  where  WDs  are  very  overloaded,  it  is  not  uncommon  for  them  to 
completely  lose  situation  awareness  all  at  once. 

•  WDs  often  have  to  solve  ill-defined  problems.  Ill-defined  problems  are  those  for  which 
there  is  no  obvious  single  solution  that  is  the  “best”  solution.  A  common  problem  of 
this  kind  is  caused  by  unknown  or  enemy  aircraft  whose  intent  is  not  clear.  For 
example,  in  tense  situations  enemy  aircraft  will  often  fly  close  to  international  borders, 
not  quite  crossing  them  but  remaining  in  threatening  positions.  WDs  have  to  monitor 
such  aircraft  constantly  to  try  to  determine  what  that  aircraft  is  going  to  do  (will  it 
become  a  threat?).  Another  problem  is  caused  when  missions  and  ROE  change  after 
take  off:  if  changes  are  not  communicated  clearly  WDs  can  often  be  uncertain  about 
what  to  do  in  encounters  with  enemy  aircraft.  When  in  doubt,  WDs  ask  the  SD  for 
clarification.  Further,  information  is  often  inaccurate  or  badly  transmitted:  Because  of 
the  high  workload,  pilots  and  other  WDs  can  make  mistakes  in  their  transmissions  of 
information,  and  the  noisiness  of  the  communication  channels  often  leads  to 
misinterpretation  of  information. 

•  Sometimes  WDs  don’t  get  the  resources  they  expect.  This  means  that  they  often  have 
to  form  new  strategies  spontaneously.  Although  missions  are  organized  so  that  WDs 
will  have  access  to  enough  resources  to  do  their  job  (preferably  meeting  threats  at 
least  one-on-one),  when  unexpected  events  occur  WDs  can  often  find  themselves  short 
or  resources. 

— Task  Performance 

•  In  principle,  all  WDs  have  access  to  all  information.  However,  there  is  often  too  much 
information  for  one  person  to  handle  (after  all,  that  is  why  WDs  operate  as  a  team), 
and  so  WDs  generally  only  attend  to  information  that  is  relevant  for  their  own  needs. 
WDs  maintain  situation  awareness  and  make  decisions  by  integrating  relevant 
information  from  the  scope,  the  radio  (information  from  pilots,  other  WDs,  the  SD, 
and  intelligence  sources),  and  the  ATO. 

•  WDs  often  change  the  expansion  on  their  scope,  periodically  zooming  out  to  do  a 
“clock  sweep”:  using  the  smallest  screen  magnification  (the  “big  picture”)  to  take 
stock  of  events  in  the  whole  AOR  and  to  look  for  any  changes  or  new  tracks. 

•  WDs  usually  use  a  set  of  relative  coordinates  for  transmitting  aircraft  positions,  using  a 
base  altitude  and  a  “bullseye”  point  on  the  screen  relative  to  which  aircraft  positions 
are  calculated.  Base  altitude  and  bullseye  points  are  preset,  and  change  from  mission 
to  mission.  This  coordinate  system  is  intended  to  prevent  the  enemy  from  interpreting 
commands  issued  by  WDs.  Remembering  to  offset  all  coordinates  is  an  added  burden 
to  WDs,  and  WDs  must  also  remember  not  to  report  the  positions  of  enemy  aircraft 
using  relative  coordinates:  knowing  their  own  absolute  positions,  the  enemy  could 
easily  work  out  where  the  base  altitude  and  bullseye  positions  are. 

•  WDs  must  maintain  efficient  “fighter  flow”.  This  involves  trading  off  between  having 
too  many  assets  airborne  and  risking  overloading  themselves,  and  having  enough 
assets  available  to  complete  the  mission.  WDs  solve  this  problem  by  planning  ahead  as 
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much  as  possible:  refueling,  scrambling  and  committing  aircraft  to  deal  with  upcoming 
events  as  efficiently  as  possible. 

•  WDs  must  prioritize  tasks,  making  sure  the  most  critical  ones  receive  the  most 
attention  and  that  each  is  revisited  at  appropriate  intervals,  yet  not  neglecting  other 
tasks.  WDs  often  use  a  strategy  called  “walking  the  clock”  to  make  sure  that  nothing 
is  neglected:  after  each  task  has  been  attended  to,  the  WD  zooms  out  and  scans  the 
screen  in  a  clockv^se  direction,  looking  for  the  next  task  under  his  control,  and  so  on 
until  the  whole  screen  has  been  scanned  and  all  tasks  serviced,  at  which  point  the  cycle 
is  repeated.  WDs  have  to  constantly  reassess  their  priorities  as  the  scenario  changes. 

— Information  Flow 

•  Since  WDs  do  not  specialize  in  a  particular  role,  each  knows  what  information  he 
needs  to  pass  on,  and  to  whom.  Information  is  usually  passed  over  Net  2,  or 
sometimes  “Net  4”,  so  the  intended  recipient  and  the  SD  can  always  hear  the  message. 
WDs  establish  exactly  how  they  will  communicate  with  each  other  by  making  verbal 
contracts  with  each  other  beforehand.  However,  in  times  of  high  activity  Net  2  and 
Net  4  are  often  saturated,  so  WDs  often  use  console  messaging,  a  kind  of  email 
between  consoles,  so  as  not  to  add  to  the  verbal  traffic.  To  avoid  confusion  and 
additional  clutter,  WDs  also  often  silently  rehearse  verbal  messages  before  speaking. 

•  Because  there  is  so  much  information  being  passed,  WDs  cannot  always  be  sure  that 
any  given  message  has  been  received:  even  if  the  recipient  hears  the  message  or  sees  it 
flash  up  on  his  screen,  he  might  not  absorb  the  information  if  he  is  very  busy.  WDs 
usually  make  contracts  with  each  other  to  acknowledge  any  messages  that  are  passed 
to  them.  If  a  WD  doesn’t  acknowledge  receipt  of  a  message,  the  sender  will  either 
repeat  it  or  somehow  gain  the  intended  recipient’s  attention,  such  as  nudging  him  on 
the  arm. 

•  WDs  constantly  communicate  with  the  pilots  they  are  controlling,  constantly  issuing 
“picture  calls”  to  update  their  pilots  with  new  mission-relevant  information,  such  as 
enemy  movements  or  other  events  happening  in  their  vicinity,  mission  changes,  and 
advance  notice  on  any  upcoming  tasks.  Information  is  usually  only  passed  to  pilots 
when  something  new  happens.  However,  sometimes  WDs  issue  picture  calls  even 
when  there  is  no  new  information,  since  pilots  are  human  and  need  to  maintain 
situation  awareness.  This  highlights  a  fundamental  difference  between  the  WD  task 
and  other  control  tasks  such  as  UAV  control,  in  which  commands  only  need  be  issued 
when  a  change  needs  to  be  effected. 

— Teamwork 

•  Missions  in  which  responsibilities  are  divided  by  geographical  area  might  proceed  with 
little  interaction  among  WDs,  as  long  as  each  WD  takes  care  of  his  own  area  and  there 
is  little  spill-over  between  lanes.  However,  when  aircraft  do  cross  between  one  lane 
and  another  it  is  imperative  for  WDs  to  negotiate  how  to  handle  that  aircraft,  a 
negotiation  that  often  involves  control  being  handed  from  one  WD  to  another. 
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•  Missions  in  which  responsibilities  are  divided  according  to  function  demand  much 
more  team  interaction.  For  example,  the  AOR  WD  has  to  temporarily  hand  control  of 
fighters  going  for  refueling  to  the  tanker  WD.  The  WD  controlling  the  tankers  might 
already  be  trying  to  coordinate  the  fueling  of  other  aircraft,  perhaps  “belonging”  to  a 
3"*  WD,  and  so  the  three  would  need  to  negotiate  an  acceptable  flieling  schedule 
according  to  the  needs  of  each  WD  and  the  aircraft  involved.  WDs  always  need  to 
keep  other  WDs  informed  of  things  that  may  affect  them:  for  example,  if  a  dogfight 
gets  dangerously  close  to  the  location  of  the  tankers,  the  AOR  WD  needs  to  tell  the 
Tanker/HVA  WD,  so  that  he  can  move  them. 

•  WDs  try  to  remain  aware  of  the  “big  picture”,  to  the  extent  it  is  possible  given  the 
demands  of  the  job,  by  monitoring  what  the  other  WDs  are  doing.  They  do  this  by 
looking  at  the  whole  area  on  the  scope,  and  also  by  listening  to  others  WDs’  radio 
frequencies.  While  it  is  difficult  for  each  WD  to  remain  aware  of  the  big  picture  in 
times  of  high  workload,  workload  is  often  distributed  unevenly  and  a  WD  who  is 
relatively  free  will  volunteer,  or  will  be  asked  by  the  SD,  to  help  another  WD  who  is 
overloaded.  For  example,  the  AOR  WD  might  ask  the  check-in  WD  to  monitor  the 
“big  picture”,  looking  out  for  new  events,  while  the  AOR  WD  is  “zoomed  in”  on  a 
tricky  intercept. 

•  Noticing  a  WD  “tunneled  in”,  focusing  on  a  small  part  of  the  airspace  using  a  large 
screen  magnification  factor  to  the  detriment  of  all  else  that  is  going  on,  is  a  sure  sign 
that  a  WD  is  overloaded  and  headed  for  trouble.  Because  WDs  share  a  common 
mental  model  of  the  task,  each  knows  the  common  problems  and  errors  associated 
with  each  role,  and  will  try  to  help  out. 

•  It  is  sometimes  more  efficient  for  a  WD  to  ask  another  WD  for  information  rather  than 
to  break  concentration  and  look  for  the  information  himself,  especially  when  asking  for 
information  regarding  assets  under  another  WD’s  control.  It  is  also  common  for  a  WD 
to  ask  another  WD  for  information  that  that  WD  has  forgotten  to  pass  on. 

•  In  crisis  situations,  such  as  when  a  WD  completely  loses  situation  awareness,  WDs 
sometimes  “seize”  other  WDs’  assets.  At  other  times  WD  teams  virtually  reconfigure 
themselves,  each  WD  taking  on  new  responsibilities  in  light  of  the  crisis  situation. 

WDs  are  able  to  do  this  since  all  receive  the  same  training,  and  the  WD  workstation 
supports  flexibility  by  allowing  all  WDs  to  control  all  assets:  to  control  an  asset,  a  WD 
usually  has  to  bring  it  up  on  his  primary  radio  frequency.  However,  each  asset  is  under 
one,  and  only  one,  WD’s  responsibility  at  any  given  time. 

— Interface  Issues 

•  The  two  dimensional  scope  displays  three  dimensional  information:  WDs  have  to  be 
constantly  aware  of  aircraft  altitudes,  the  third  dimension  of  airspace  that  is  not 
represented  graphically  on  the  scope.  WDs  can  find  out  aircraft  altitudes  from  the  TDs 
or  by  using  an  option  that  displays  the  track  altitudes  as  part  of  their  symbology. 

•  WDs  need  to  work  at  a  number  of  different  levels  of  screen  magnifications:  WDs  can 
use  their  scopes  to  zoom  in  on  a  portion  of  the  airspace  in  order  to  closely  control  and 
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monitor  aircraft,  or  zoom  out  to  look  at  the  “big  picture”.  However,  there  are  costs 
associated  with  both  of  these  views;  while  zoomed  in  on  a  small  area  the  WD  cannot 
see  what  is  going  on  in  other  areas  of  the  airspace,  and  while  zoomed  out  WDs  cannot 
make  precise  bearing  and  range  measurements  or  select  specific  tracks  very  easily,  due 
to  screen  resolution  limitations.  These  limitations  are  important  determinants  of  WD 
behavior:  the  latter  limitation  requires  that  WDs  spend  a  lot  of  their  time  zoomed  in, 
enlisting  other  WDs  and  using  the  radio  to  maintain  awareness  of  the  big  picture. 

•  Aircraft  often  fly  so  close  together  that  they  appear  on  the  scope  as  a  single  track.  At 
other  times,  even  though  there  are  in  fact  two  tracks  they  may  be  so  close  together 
that  they  look  like  one.  This  can  sometimes  cause  WDs  to  underestimate  the  strength 
of  the  enemy.  On  the  other  hand,  friendly  aircraft  usually  fly  in  flights  of  two  or  four, 
with  the  WD  only  talking  to  the  lead  pilot.  The  advantage  of  this  is  that  the  WD  can 
treat  a  number  of  aircraft  as  a  single  unit,  while  the  disadvantage  is  that  he  must  often 
remember  how  many  aircraft  are  represented  by  a  single  track  (or  what  looks  to  be  a 
single  track). 

•  WDs  “know  their  switches”:  although  the  WD  console  is  superficially  complex,  WDs 
can  perform  most  of  the  common  operations  with  little  effort  wasted  on  figuring  out 
which  button  to  press. 

•  Only  a  small  number  of  basic  actions  are  required  for  WDs  to  perform  their  job 
efficiently.  The  apparent  difficulty  of  the  WD  job  derives  from  the  complexity  of  the 
situations  that  WDs  face,  the  amount  of  information  they  have  to  process,  and  the  fact 
that  WDs  may  have  to  perform  many  activities  at  once. 

•  Although  WDs  are  trained  to  exercise  precise  control  over  aircraft,  in  reality  fighters 
have  their  own  radar  that,  over  short  ranges,  is  more  precise  than  the  AW ACS  radar. 
In  real  missions,  the  role  of  the  WD  is  to  guide  fighters  to  within  range  of  their  target, 
be  it  a  hostile  aircraft  or  a  tanker,  after  which  the  WD  monitors  the  situation  and  keeps 
the  pilot  informed  of  events  he  might  miss.  For  example,  in  a  dogfight  the  WD  would 
warn  the  pilots  of  any  “spitters”  (enemy  aircraft  that  disengage  and  split  off  from  the 
main  battle). 

•  The  AW ACS  radar  has  a  12  second  sweep,  meaning  that  the  scope  information  is 
updated  every  12  seconds  and  so  can  become  up  to  12  seconds  out  of  date  during  the 
time  a  WD  has  to  make  a  decision.  WDs  have  to  be  aware  of  this,  and  extrapolate  the 
positions  of  aircraft.  This  adds  to  the  WDs’  workloads  and  causes  errors,  but  does  not 
affect  strategies  for  dealing  with  external  events.  Fighter  radars  update  more  quickly 
than  the  AW ACS  radar,  and  so  when  a  fighter  has  a  target  on  its  radar  the  WD  will 
rely  on  the  fighter  pilot’s  radio  transmissions  for  up-to-date  reports  of  when  a  target 
has  changed  heading. 

•  WDs  can  spend  a  disproportionate  amount  of  time  “tracking”  objects  whose 
symbologies  have  become  detached.  However,  in  a  soon-to-be  implemented  updated 
AW  ACS  system,  this  problem  (i.e.  sub-task)  has  been  virtually  eliminated. 

•  The  WD’s  job  is  affected  by  the  weather:  WDs  have  to  exercise  different  levels  of 
control  over  their  assets  depending  on  the  weather.  Regulations  state  that  air  refueling 


can  only  be  done  under  conditions  in  which  aircraft  can  see  each  other,  and  so  WDs 
often  need  to  work  with  aircraft  to  find  it  clean  air. 


Specific  Events 

In  this  section  we  describe  how  WDs  deal  with  particular  events  and  situations. 
Individual  events  always  occur  in  a  particular  mission  context,  and  a  WD’s  response  to 
any  event  depends  on  all  else  that  is  happening  at  that  particular  time.  However,  through 
sampling  scenarios  and  critical  incidents  in  our  interviews,  it  is  possible  to  build  up  a 
context-free  picture  of  how  particular  events  are  dealt  with,  and  an  understanding  of  the 
contextual  factors  that  determine  exactly  how  an  event  will  be  responded  to.  Findings 
from  this  kind  of  analysis  can  be  used  both  for  scenario  development  (deciding  which 
events  to  use  in  order  to  study  particular  task  characteristics),  and  for  suggesting 
performance  metrics  based  on  task  processes,  not  just  mission  outcomes.  We  will  outline 
some  common  events  and  situations  faced  by  WDs,  and  explain  how  they  deal  with  them. 

In  describing  events,  for  the  most  part  we  assume  a  team  composed  of  a  Check-in 
WD,  an  AOR  WD,  and  a  Tanker/HVA  WD. 

— Routine  Events 

Each  mission  is  precisely  scheduled.  During  briefing,  WDs  are  given  copies  of  the 
Air  Tasking  Order  (ATO),  which  details  the  flight  schedules  of  all  aircraft  that  will  be 
flying  on  that  team’s  “watch”.  Aircraft  are  scheduled  to  take  off  at  a  particular  time,  to  fly 
a  designated  route,  and  to  land  at  a  particular  time.  Air  refueling  is  also  pre-scheduled. 
Missions  are  generally  configured  not  to  over-tax  the  WD  team,  and  if  a  mission  goes 
according  to  plan  most  events  are  manageable,  with  each  team  member  having  routine 
responsibilities  for  keeping  the  other  WDs  abreast  of  events  as  follows: 

•  The  Check-in  WD  needs  to  identify  and  attach  symbology  to  any  aircraft  taking  off, 
checking  whether  they  are  as  frag’d  (as  described  in  the  ATO).  Once  this  is  done,  he 
“pushes”  the  aircraft  to  the  AOR  or  Tanker/HVA  frequency  (depending  on  what  the 
aircraft  is),  and  tells  the  appropriate  WD  that  this  has  happened  and  whether  or  not  the 
aircraft  is  as  frag’d.  The  Check-in  WD  also  takes  control  of  aircraft  that  are  going 
back  to  base. 

•  The  AOR  WD  needs  to  tell  the  SD  when  anything  out  of  the  ordinary  happens  in  the 
AOR,  such  as  deviations  from  the  mission  plan.  He  tells  the  Tanker/HVA  WD  when 
an  aircraft  is  coming  to  a  tanker  for  refueling,  and  tells  the  Check-in  WD  when  an 
aircraft  is  leaving  the  AOR. 

•  The  Tanker/HVA  WD  monitors  the  refuelings  and  the  positions  of  high  value  assets 
(HVA)  such  as  reconnaissance  aircraft.  He  informs  the  AOR  WD  when  an  aircraft 
leaves  the  tanker  and  re-enters  the  AOR,  or  when  something  happens  with  the  HVAs 
that  is  not  on  the  ATO.  He  also  tells  the  Check-in  WD  when  any  tankers  or  HVAs  are 
going  back  to  base. 
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•  The  SD  monitors  the  other  WDs,  and  keeps  them  informed  of  any  planned  mission 
changes  or  changes  to  the  ATO  that  he  receives  from  the  MC  or  Grround  Control. 

Each  WD  is  responsible  for  informing,  as  appropriate,  the  other  WDs,  and  the  SD 
when  anything  happens  that  is  not  exactly  according  to  plan  (or  if  a  planned  event  doesn’t 
happen).  The  SD  needs  to  be  kept  informed  of  almost  all  unplanned  events.  For  example, 
the  Check-in  WD  would  have  to  tell  both  the  Tanker/HVA  WD  and  the  SD  if  a  scheduled 
tanker  failed  to  appear.  The  Tanker/HVA  WD  would  need  to  tell  the  AOR  WD  that  the 
fuel  will  not  be  available  as  frag’d,  and  it  would  be  the  SD’s  responsibility  to  find  out  why 
the  tanker  did  not  materialize  and  to  arrange  for  a  replacement.  This  rule  holds  for  both 
changes  in  events  and  changes  in  the  timing  of  events:  for  example,  the  AOR  WD  would 
need  to  tell  the  Check-in  WD  if  an  aircraft  was  not  going  to  check  out  at  the  scheduled 
time. 


Unsurprisingly,  the  most  interesting  team  interactions  occur  when  crisis  or  difficult 
situations  develop,  going  beyond  missions  in  which  the  only  unexpected  events  are 
deviations  from  the  ATO.  Wien  WDs  encounter  unknown  or  hostile  aircraft,  the  main 
priority  is  to  defend  the  airspace  and  fiiendly  aircraft.  WDs  become  much  more 
opportunistic:  they  need  to  use  the  assets  they  have  efficiently,  refueling  and  RTB’ing 
either  when  absolutely  necessary  or  when  it  is  most  convenient,  and  scrambling  new 
aircraft  when  needed.  Coordination  becomes  paramount,  since  each  WD  has  his  own 
needs.  For  example,  the  AOR  WD  ne^s  to  make  sure  that  there  are  enough  fighters  on 
station,  and  the  Tanker/HVA  WD  needs  to  spontaneously  form  a  refiieling  schedule.  Both 
WDs  need  to  work  together  to  make  sure  that  things  happen,  all  the  while  keeping  the  SD 
and  the  Check-in  WD  informed  of  what  is  going  on  in  order  to  minimize  the  chances  of 
any  “surprises”. 

— Non-Routine  Events 

Here  we  describe  some  of  the  non-routine  events  that  were  discussed  during  our 
interviews.  While  this  is  not  a  comprehensive  list  of  all  events  that  can  possibly  happen  in 
all  possible  contexts,  it  gives  a  sampling  of  the  cognitive  demands  placed  on  WD  teams 
and  the  decisions  that  WDs  need  to  make. 

•  Hostile  threat 

The  AOR  WD  is  responsible  for  addressing  any  threats  that  appear.  When  a  hostile 
or  potentially  hostile  track  appears  in  the  AOR,  the  AOR  WD  first  needs  to  inform  the 
SD.  The  SD  and  the  WD  often  work  together  to  decide  what  to  do  about  it,  with  the  SD 
coordinating  with  the  Mission  Controller  (MC).  A  good  WD  will  approach  the  SD  with  a 
solution  as  well  as  a  problem,  and  SDs  will  often  act  upon  WDs’  recommendations.  Once 
it  has  been  determined  that  a  track  is  a  threat,  the  WD  and  the  SD  figure  out  how  many 
aircraft  are  needed  to  address  the  threat,  then  figure  out  where  these  will  come  from.  The 
first  place  to  look  is  to  aircraft  currently  in  the  £ur,  taking  into  account  what  else  they  may 
be  doing,  their  capabilities,  and  their  fuel  and  armament  status.  Sometimes  it  is  necessary 
to  scramble  new  aircraft,  typically  done  through  the  SD:  the  SD  will  coordinate  with  the 
MC  and  Ground  Control  to  get  the  new  aircraft,  then  tell  the  Check-in  WD  to  expect  the 
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new  aircraft.  As  the  situation  unfolds,  the  AOR  WD  directs  the  fnendly  aircraft 
appropriately.  How  he  directs  will  depend  on  the  types  of  aircraft  involved  (e.g.,  “are  the 
hostile  aircraft  faster  than  the  fnendly  aircraft?”),  and  the  current  ROE  (e  g.,  whether  to 
shoot  or  head  the  hostiles  off).  Sometimes,  if  the  hostile  aircraft  has  not  actually 
committed  a  hostile  act  (such  as  shooting,  bombing,  or  “painting”  an  aircraft  with  its 
weapons  radar),  the  fiiendly  aircraft  may  have  to  shadow  the  hostiles,  following  them 
closely  until  they  cease  to  be  a  threat  or  further  action  needs  to  be  taken.  This  requires 
close  monitoring,  and  working  through  the  SD  to  decide  when  further  action  needs  to  be 
taken. 

•  Hostile  track  menacing  the  border 

In  sensitive  situations,  hostile  aircraft  often  fly  close  to  borders,  not  quite  crossing 
them,  to  “test”  the  fnendly  defense.  It  is  the  AOR  WD’s  responsibility  to  make  sure  that  a 
menace  does  not  become  a  threat.  As  with  a  hostile  threat,  the  AOR  WD  needs  to  tell  the 
SD  what  is  going  on,  and  assign  fnendly  fighters  to  monitor  the  menace.  This  situation 
requires  constant  monitoring,  and  the  WD  and  the  SD  need  to  decide  when  the  hostile 
ceases  to  menace  and  becomes  a  threat. 

•  Unknown  track 

The  surveillance  team  is  primarily  responsible  for  identifying  and  attaching 
symbology  to  tracks  that  are  not  part  of  the  mission,  usually  those  originating  outside  of 
friendly  territory.  However,  sometimes  unknown  tracks  are  not  immediately  identified. 

The  AOR  WD  must  first  tell  the  SD  that  there  is  an  unknown  track,  then  monitor  the 
track’s  progress  until  it  gets  into  a  threatening  position.  It  is  then  treated  as  a  threat.  The 
ROE  usually  specify  that  the  track  has  to  be  identified,  and  there  are  usually  specific 
criteria  for  what  counts  as  a  valid  identification  (e.g..  Visual  ID,  Beyond- Visual-Range 
ID).  The  AOR  WD  will  usually  send  fighters  out  to  ID  it  and,  depending  on  the  response, 
will  either  treat  it  as  a  threat  or  leave  it  alone. 

•  Defector 

There  are  pre-specified  actions  that  defectors  must  take  in  order  to  be  taken 
seriously.  Defectors  must  broadcast  an  identification  code,  and  must  fly  slowly  with 
landing  gear  down,  weapons  checked  “safe”,  and  fire  control  radar  turned  off.  They  must 
broadcast  their  intent  on  the  Guard  (common  international)  channel.  Due  to  the  potential 
for  trickery,  the  AOR  WD  usually  sends  some  of  his  fighters  to  investigate  the  defector 
and  escort  it  to  a  base.  This  means  essentially  taking  some  fnendly  fighters  out  of  the 
mission,  and  so  the  AOR  WD  and  the  SD  have  to  decide  whether  to  try  to  scramble  new 
aircraft  as  replacements. 

•  Hostile  track  splits  up 

Aircraft  seldom  fly  alone,  and  sometimes  aircraft  fly  so  close  together  that  on  the 
AW  ACS  radar  two  aircraft  can  look  like  one  track.  Aircraft  will  often  fly  close  to  one 
another  in  order  to  fool  the  AW  ACS  radar,  then  split  up  at  the  last  minute,  confronting  the 
AOR  WD  with  twice  as  many  tracks  as  he  previously  thought.  In  such  cases  there  is 
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pressure  on  the  AOR  WD  to  respond  quickly  in  order  to  prevent  any  threats  penetrating 
the  friendly  airspace.  The  AOR  WD  may  respond  in  kind,  splitting  up  his  own  flights  into 
separate  tracks  (which  requires  permission  from  the  SD),  or  may  either  direct  some 
additional  aircraft  in  from  another  area,  scramble  some  new  aircraft,  or  prematurely  take 
some  aircraft  off  the  tanker  if  there  is  a  real  problem. 

•  Shooting  down  a  hostile  aircraft 

Depending  on  the  ROE,  fighters  often  need  to  ask  for  permission  to  shoot.  When  a 
pilot  asks  a  WD  for  an  “open  right  to  kill”  (ORK),  the  request  is  relayed  via  the  SD  to  the 
Ground  Control,  then  relayed  back  to  the  pilot.  After  permission  has  been  granted,  the 
WD  needs  to  monitor  the  situation  closely,  listening  for  the  fiiendly  pilot  to  announce  that 
he  has  shot  down  the  hostile  track  or  for  signs  that  the  hostile  track  has  shot  down  the 
friendly  track.  Regardless  of  the  outcome,  the  WD  needs  to  tell  the  SD,  who  then  informs 
the  MC  and  Ground  Control  of  what  happened.  If  the  hostile  was  shot  down,  the  WD 
needs  to  provide  guidance  to  the  pilot  on  what  to  do  next,  while  if  the  friendly  aircraft  has 
been  shot  down,  a  Search  and  Rescue  (SAR)  mission  needs  to  be  initiated. 

•  Dogfight 

When  aircraft  get  close  together  in  a  dogfight,  there  is  not  much  that  the  WD  can 
do  to  help,  since  at  that  point  the  fighter  radars  are  usually  more  capable  than  the  AW ACS 
radar.  The  role  of  the  WD  is  to  monitor  the  fight,  waiting  for  kill  calls,  bail-out  calls,  and 
watching  for  “spitters”  (hostile  aircraft  that  “sneak  out”  of  the  dogfight  unobserved). 

•  Strike  Package 

The  AOR  WD  needs  to  make  sure  that  there  is  a  clear  path  for  the  strike  package 
(bombers  and  their  escorts)  to  go  in.  He  will  use  his  fighters  to  neutralize  any  threats  that 
are  in  the  way.  Since  strike  packages  are  usually  scheduled,  the  mission  will  be  set  up  so 
that  WDs  have  enough  resources  to  do  this  while  still  being  able  to  cover  the  rest  of  the 
airspace. 

—How  Events  Generate  Teamwork 

As  mentioned  before,  some  team  interactions  are  routine,  based  on  how 
information  needs  to  be  passed  on.  There  are  few  events  which,  in  themselves,  generate 
teamwork,  except  for  crisis  situations,  where  all  WDs  tend  to  work  together  to  solve  the 
problem.  However,  the  most  interesting  team  interactions  arise  opportunistically.  Many 
interactions  arise  when  a  WD  is  over-tasked,  when  another  may  jump  in  and  lend  a  hand. 
We  observed  some  real  examples  of  how  this  can  happen  in  the  critical  incident  interviews, 
and  the  main  critical  incidents  are  given  in  Appendix  B.  Here  are  some  of  the  more 
common  situations  that  give  rise  to  team  interactions: 

•  Too  many  aircraft  checking  in  at  once 

During  the  early  part  of  the  mission,  the  Check-in  WD  can  often  become 
overwhelmed.  Yet  it  is  imperative  that  he  keep  up  with  the  work.  Since  the  mission  proper 
won’t  have  really  started,  the  AOR  WD  usually  helps  out  with  check-in  duties.  If  aircraft 


31 


are  taking  off  from  more  than  one  airfield,  the  Check-in  WD  will  deal  with  aircraft  taking 
off  from  one  airfield  while  the  AOR  WD  will  take  care  of  the  other. 

•  Too  many  aircraft  in  the  AOR  at  once 

Later  on  in  the  mission  the  reverse  is  true:  the  Check-in  WD  has  little  to  do,  while 
the  AOR  WD  can  be  overwhelmed.  Accordingly,  the  Check-in  WD  often  helps  out  the 
AOR  WD,  performing  tasks  such  as  tracking  aircraft  and  monitoring  the  “big  picture” 
while  the  AOR  is  otherwise  busy. 

•  Incidents  in  different  parts  of  the  scope 

Sometimes  the  AOR  WD  has  to  deal  with  incidents  in  different  parts  of  the  scope. 
This  can  be  difficult  because  it  is  often  necessary  to  “zoom  in”  on  each  incident  in  order  to 
see  what  is  going  on  in  sufficient  detail,  and  so  it  is  often  not  possible  to  monitor 
numerous  incidents  at  the  same  time.  The  AOR  WD  will  often  enlist  another  WD  to  help 
out. 


•  Tracking  a  helicopter 

Helicopters  are  extremely  difficult  to  track,  because  they  fly  at  a  low  altitude  and 
at  a  slow  speed.  Because  of  this,  they  often  disappear  from  the  radar,  and  it  is  easy  for  a 
busy  WD  to  forget  that  they  are  there.  To  make  matters  worse,  helicopters  are  often 
owned  by  other  agencies,  such  as  the  Army  or  the  United  Nations,  and  so  often  do  not 
check  in  with  the  WDs.  This  can  cause  them  to  appear  suddenly  in  the  middle  of  the 
screen,  as  a  “pop-up”  track  (see  below).  A  WD  will  often  recruit  a  less-busy  WD  to  track 
a  helicopter,  since  it  requires  too  much  concentration  for  someone  who  is  busy  elsewhere. 

•  Pop-up  track 

Sometimes  an  unidentified  track  will  suddenly  appear  on  the  radar,  or  fnendly 
aircraft  will  suddenly  “lock  on”  to  a  target  that  is  not  on  the  AW  ACS  radar.  There  is  often 
a  time  constraint  in  these  situations,  since  the  “pop-up”  can  appear  anywhere,  including  in 
friendly  airspace.  If  the  pop-up  is  a  hostile  aircraft  it  needs  to  be  dealt  with  right  away; 
however,  it  is  important  not  to  rush  into  action  since  the  consequences  of  a  friendly  fire 
incident  are  very  grave.  The  AOR  WD,  or  whoever  first  notices  the  pop-up,  needs  to  first 
inform  all  the  other  WDs  and  the  SD.  The  AOR  WD  and  the  SD  need  to  coordinate 
friendly  fighters  in  the  area,  and  the  first  priority  is  to  identify  the  unknown  track.  This  is 
usually  done  by  fnendly  fighters  working  closely  with  the  AOR  WD,  while  the  other  WDs 
will  work  to  determine  if  there  are  any  fnendly  assets  that  they  could  have  missed  (such  as 
helicopters  that  might  have  flown  below  the  radar),  or  aircraft  returning  to  the  AOR 
without  checking  back  in. 

•  Friendly  aircraft  shot  down 

When  a  friendly  aircraft  goes  down,  the  WD  needs  to  initiate  a  Search  and  Rescue 
(SAR)  mission  to  recover  any  survivors,  and  coordinate  any  support  aircraft  that  are 
needed.  The  AOR  WD  usually  knows  the  location  of  the  downed  aircraft,  but  he  has  to 
pinpoint  it  exactly  in  order  to  run  an  efficient  SAR  mission.  If  possible,  he  will  have  some 
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nearby  aircraft  fly  over  the  crash  site.  The  SD  often  helps  to  launch  the  SAR  package,  and 
control  of  this  will  often  be  given  to  the  least  busy  WD, 

•  Spill-over  between  lanes 

When  there  is  spill-over  of  hostile  tracks  or  a  battle  from  one  lane  to  another,  the 
WDs  involved  need  to  negotiate  how  to  deal  with  it  between  themselves;  this  could 
involve  passing  control  of  friendly  fighters  from  one  WD  to  another,  or  one  WD 
temporarily  working  within  the  other’s  lane. 

•  Shortage  of  resources 

When  a  WD  runs  short  of  resources,  he  usually  works  through  the  SD  to  get  more 
aircraft  airborne,  or  with  the  Tanker/HVA  controller  to  refuel  the  aircraft  he  does  have.  In 
some  cases,  such  as  when  using  the  “lane  defense”  configuration,  a  WD  can  negotiate  with 
another  WD  to  “borrow”  aircraft.  For  example,  if  an  aircraft  runs  out  of  fuel  when 
shadowing  a  hostile  track,  the  WD  might  negotiate  to  get  the  nearest  set  of  fighters  to 
cover,  regardless  of  who  is  controlling  them. 

•  Refueling  in  bad  weather 

When  refueling  in  bad  weather,  the  tankers  need  to  find  “clean  air”  in  which  to 
work.  This  can  require  considerable  coordination  with  the  Tanker/HVA  WD,  who  will 
often  need  to  enlist  the  help  of  another  WD  to  oversee  other  tasks  while  he  works  with  the 
tankers. 

•  WD  loses  situation  awareness 

When  one  WD  gets  overloaded  and  loses  situation  awareness,  it  is  up  to  the  whole 
team  to  cover  him  and  help  him  get  back  on  track.  The  SD  usually  steps  in,  dividing 
control  of  the  lost  WD’s  resources  between  the  other  WDs,  and  working  to  help  him 
rebuild  situation  awareness.  A  common  way  to  do  this  is  to  zoom  out  the  scope  in  order 
to  see  the  big  picture,  and  to  listen  to  the  radio  for  while  to  gradually  figure  out  what  is 
going  on,  before  resuming  control. 

SYNTEAM  DESIGN 


A  major  challenge  in  designing  an  STTE  is  deciding  exactly  how  to  simplify  the 
real  world  task  without  significantly  altering  its  main  characteristics.  It  would  be  very  easy 
to  oversimplify  the  task  inadvertently.  For  example,  real  world  air  refiieling  takes  quite  a 
lot  of  time.  If  air  refueling  in  the  STTE  is  made  extremely  quick  and  simple,  then  decisions 
on  whether  or  not  to  refuel  an  aircraft  might  be  taken  much  more  lightly.  In  addition, 
many  real  world  task  characteristics  simply  add  cognitive  overhead  to  the  task.  For 
example,  the  AW  ACS  radar  scope  updates  every  12  seconds,  which  causes  screen 
information  to  become  outdated  even  while  audio  information  from  pilots  is  continually 
updated.  This  does  not  change  the  fitndamental  characteristics  of  the  decision  to  be  made, 
but  nevertheless  adds  to  the  overall  workload  and  can  cause  errors  and  confusion, 
especially  if  the  (outdated)  screen  information  conflicts  with  the  (current)  audio 
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information.  Whether  to  include  such  characteristics  into  SynTEAM  is  a  judgment  call, 
which  depends  on  the  kinds  of  studies  SynTEAM  is  intended  to  enable. 

Summary  of  CTA  Findings 

To  recap,  here  are  the  most  prominent  findings  from  the  CTA. 

Task  Characteristics 

•  Dynamic,  dense,  audio-visual  environment,  requiring  multitasking  and  making  heavy 
perceptual  demands 

•  Varied  duties  and  responsibilities  and  a  variety  of  mission  objectives 

•  Scheduled,  rule-bound  routine  missions,  but  unexpected  events  and  emergencies 
causing  deviations  fi'om  the  schedule  and  changes  to  the  rules 

•  Limited  resources 

•  Small  number  of  basic  operations 

Team  Characteristics 

•  Hierarchiceil  team  structure,  with  SD  or  higher  making  most  of  the  strategic  decisions 

•  WDs  flexibly  divide  responsibilities,  producing  differential  workloads  throughout  the 
mission 

•  Overlapping  mental  and  situation  models 

Team  Strategies  and  Operating  Principles 

•  Provide  universal  access  to  information 

•  Integrate  information  from  scope  and  radios 

•  Remain  aware  of  the  big  picture:  Use  different  scope  magnifications  and  “clock 
sweeps” 

•  Prioritize  tasks  effectively 

•  Pass  on  relevant  information  to  other  WDs  and  pilots,  making  sure  it  gets  there 

•  Monitor  each  other 

•  Help  each  other  out  when  the  need  and  opportunity  arises  (opportunistic  cooperation) 

Major  Errors  and  Problems 

•  Information  overload,  leading  to  loss  of  situation  awareness 

•  Failures  to  coordinate  or  pass  on  information 

•  Errors  in  passing  information 

•  Failure  to  interpret  or  absorb  information  correctly 
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Data  Capture  and  Performance  Metrics 

SynTEAM  should  have  some  basic  features  for  post-hoc  analysis  of  team  behavior: 

•  Mission  reconstruction:  SynTEAM  should  be  able  to  replay  a  mission,  recreating  how 
aircraft  moved  around  the  screen,  and  signifying  in  some  way  when  WDs  performed 
actions  (such  as  presenting  WD  actions  in  a  text  box).  Replay  should  be  synchronized 
across  WD  stations.  If  the  replay  facility  is  not  practical  to  program,  SynTEAM  should 
at  least  have  a  capability  similar  to  C^-STARS,  with  the  screen  being  sampled  and 
dumped  to  a  file  periodically,  and  a  “log  file”  being  kept,  recording  all  WD  actions  and 
scenario  events.  Another  option  is  to  use  a  multiplexer  to  capture  the  screens  of  each 
WD  station  and  record  them  onto  a  videotape.  If  this  is  done,  the  video  should  be 
time-stamped  to  allow  correlation  with  the  log  file. 

•  Voice  data:  SynTEAM  WDs  should  wear  microphones  to  capture  voice  data. 
Depending  on  how  communication  with  aircraft  is  implemented,  each  WD’s  voice 
could  be  captured  on  a  different  audio  track.  There  should  be  some  way  to 
synchronize  the  voice  data  with  the  scenario  events  and  WD  actions  data,  however 
they  are  captured  (see  point  above). 

•  Experimenter  station:  a  useful  feature  would  be  an  option  for  an  experimenter  console, 
from  which  a  researcher  could  monitor  the  mission  progress,  displaying  each  WD’s 
scope  in  a  window  on  a  large  screen  and  allowing  the  experimenter  to  annotate  and 
make  notes. 

Many  performance  metrics  have  been  refined  using  the  C^-STARS  simulator,  and 
most  of  these  should  be  transferable  to  SynTEAM.  However,  many  of  these  are 
“outcome”  measures,  such  as  kill  ratio  and  total  number  of  refuelings.  SynTEAM  should 
also  capture  some  process  measures.  Process  measures  will  indicate  whether  WDs  and 
WD  teams  are  exhibiting  the  hallmarks  of  good  and  poor  teams.  These  measures  can  be 
related  to  outcome  measures  to  determine  which  processes  are  most  important  for  good 
team  performance.  Here  are  some  ideas  for  process  measures  that  warrant  further 
development: 

•  Situation  awareness  probes:  many  existing  S5mthetic  tasks  measure  situation  awareness 
by  stopping  the  action,  blocking  out  the  screen  and  asking  questions.  Since  this  is  such 
a  popular  method,  SynTEAM  should  also  have  this  capability.  An  interesting  variant 
on  this  would  be  to  be  able  to  block  out  part  of  the  screen:  research  on  air  traffic 
controllers  has  shown  that  they  do  not  group  aircraft  by  location,  but  by  common 
purpose  (Schlager,  Means,  and  Roth,  1990);  partially  blocking  out  the  screen  would 
allow  flexibility  in  researching  this  issue. 

•  “On-line  CTA”:  when  performing  a  CTA,  we  frequently  use  the  questions  “why?”  to 
probe  interviewees  for  the  reasons  and  decisions  underlying  their  actions.  It  is  perhaps 
the  most  important  CTA  probe,  yielding  not  only  the  motivations  for  interviewees’ 
actions,  but  also  their  interpretations  of  the  situation  and  the  results  of  previous 
actions.  In  SynTEAM,  it  might  be  possible  to  capture  at  least  some  of  this  information 
as  the  task  unfolds.  The  system  could  ask  a  WD  to  “say  reason”  after  performing  an 
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action,  requiring  him  to  explain  why  he  performed  an  action.  Collecting  CTA  data  in 
such  a  way  would  present  a  huge  advantage  over  collecting  data  retrospectively,  as  is 
usually  done  with  dynamic  tasks,  and  such  a  requirement  would  even  be  ecologically 
valid,  since  pilots,  SDs,  and  MCs  often  question  WDs  in  such  a  way.  It  should  be 
possible  to  toggle  this  feature  on  and  off. 

•  Secondary  tasks:  scenarios  should  include  some  low  priority  tasks  that  are  not  critical, 
yet  which  require  attention  and  action,  and  on  which  WDs’  performance  can  be  easily 
measured.  A  number  of  these  low-priority  tasks  could  form  a  library  of  secondary 
tasks,  and  performance  on  these  tasks  should  be  correlated  with  workload:  the  less 
busy  a  WD  is,  the  more  attention  he  can  pay  to  the  secondary  task.  An  example  of  a 
secondary  task  would  be  controlling  aircraft  on  non-dangerous,  non-strategic  missions 
in  a  corner  of  the  AOR  not  affected  by  the  rest  of  the  scenario.  One  idea  would  be  to 
control  a  supply  aircraft  making  periodic  drops  over  designated  spots,  the  WD  would 
have  to  direct  the  aircraft  to  the  spot,  and  issue  a  “drop”  command  when  it  is  reached. 
The  dependent  measures  could  be  accuracy  of  drops  and  drops  missed.  Another  idea  is 
to  have  WDs  control  a  training  mission,  keeping  the  pilots  away  from  the  combat 
zone. 

•  Communication  measures:  WDs  report  that  a  better  team  will  interact  less:  this  is 
because  WDs  share  mental  and  situation  models,  and  teams  with  good  models  know 
what  is  going  on  and  how  information  needs  to  be  passed,  reducing  the  need  for  WDs 
to  query  each  other.  Other  WDs  report  that  they  use  the  electronic  messaging  system 
when  the  voice  channels  are  saturated.  Measures  based  on  the  amount  of 
communication  of  each  type  could  be  useful  as  indicators  of  mental  model  overlap, 
although  further  details  will  have  to  be  worked  out  when  the  basic  SynTEAM  design 
has  been  finalized. 

•  Keeping  the  “big  picture”:  WDs  stress  the  importance  of  staying  aware  of  all  that  is 
going  on  in  the  AOR.  However,  they  often  get  sidetracked,  becoming  preoccupied 
with  a  single  task  while  ignoring  others.  SynTEAM  should  keep  track  of  how  often 
WDs  zoom  out  their  scopes  to  look  at  the  big  picture,  and  how  often  they  stay 
zoomed  in  on  a  single  area. 

•  Keeping  aircraft  informed:  pilots  need  to  be  kept  informed  of  how  scenarios  develop, 
since  their  own  radars  have  limited  scope  and  range.  WDs  give  pilots  picture  calls 
when  the  position  or  heading  of  a  target  changes,  and  sometimes  just  to  reestablish 
contact.  SynTEAM  could  measure  the  time  elapsed  between  a  change  in  a  target 
heading  and  the  WD  directing  his  aircraft  in  response  (as  mentioned  before,  constantly 
changing  a  target’s  heading  would  be  one  way  of  keeping  a  WD  tunneled  in  on  a 
single  task).  Another  idea  is  to  have  aircraft  request  a  picture  call,  either  verbally  or 
visually  (say,  through  a  change  in  symbology).  This  could  happen  when  the  WD  has 
ignored  its  aircraft  for  a  set  amount  of  time  or,  perhaps,  when  the  WD’s  aircraft 
detects  another  aircraft  to  which  it  is  not  committed.  SynTEAM  would  measure  the 
time  elapsed  between  the  aircraft  request  and  the  WD’s  response. 

•  Compensating  for  workload:  WDs  become  busy  at  different  times,  and  in  a  good  team 
less-busy  WDs  will  step  in  to  help  those  who  are  overloaded.  SynTEAM  could 
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measure  how  busy  each  WD  is;  based  upon  the  number  of  tracks  under  his  control,  the 
number  of  actions  issued  per  unit  time  and/or  the  types  and  amounts  of  activities  of 
tracks  under  his  control.  This  ongoing  measure  of  differential  workload  could  be 
related  to  other  measures  of  how  WDs  share  tasks  or  help  each  other  out  (see  below). 

•  Sharing  tasks  and  helping;  WDs  help  each  other  by  taking  over  or  monitoring  tasks 
that  are  not  their  own.  SynTEAM  could  capture  this  by  looking  at  the  extent  to  which 
WDs  perform  operations  on  tracks  that  are  not  their  own,  or  zoom  in  on  tasks  that  are 
not  their  own.  One  idea  would  be  to  design  SynTEAM  so  that  WDs  can  monitor  their 
own  tasks  effectively  only  by  zooming  in  on  them.  In  this  case,  zooming  in  on 
someone  else’s  task  (and  losing  sight  of  ones  own  AOR)  would  unambiguously  signal 
cooperative  monitoring  and/or  work-sharing.  Scenarios  would  have  to  be  designed 
with  WDs  assigned  to  different  parts  of  a  campaign  geography  that  was  too  big  to 
monitor  while  zoomed  out. 

•  Prioritization:  appropriately  prioritizing  tasks  is  a  major  part  of  the  WD  job.  If  tasks 
are  pre-prioritized  by  experimenters,  a  measure  of  how  much  attention  is  paid  to  each 
task  can  be  compared  to  this  “ideal”  prioritization. 

In  addition  to  the  above  broadly  conceptual  process  measures,  some  simpler 
smaller-scale  generic  measures  may  be  informative  and  easy  to  record: 

•  Time  taken  to  respond  to  various  events:  heading  changes,  pilot  requests,  “spitters” 
(hostile  aircraft  breaking  away  from  a  dogfight),  downed  aircraft 

•  Time  taken  to  respond  to  messages 

•  Amount  of  time  a  hostile  aircraft  is  left  uncovered  after  it  passes  a  critical  point 
(“commit  line”) 

•  Number  of  times  control  of  aircraft  is  passed  between  WDs 

•  Errors  with  respect  to  ROE 

•  Accuracy  of  information  passed  between  WDs 

•  Whether  WDs  correctly  acknowledge  messages  passed  to  them 

•  W/hether  the  SD  is  advised  when  something  out  of  the  ordinary  happens 

Most  of  these  measures  should  be  viewed  in  light  of  the  total  amount  of  activity 
going  on  in  that  time  period,  so  that  processes  can  be  tracked  in  relation  to  workload. 
These  measures  obviously  need  to  be  refined:  we  suggest  that  the  early  versions  of 
SynTEAM  output  a  log  file  of  all  events  and  actions  that  occur  during  a  mission,  as  does 
the  C^-STARS  simulator.  These  log  files  could  then  be  used  as  raw  data  from  which  to 
construct  and  refine  process  measures.  Another  possibility  worth  considering  for  capturing 
data  on  the  focus  of  WDs’  attention  is  to  add  eye-tracking  capability.  That  way,  it  would 
be  possible  to  determine  where  a  WD’s  focus  lies  independently  of  whether  or  not  he  is 
zoomed  in  on  an  area  of  the  screen. 
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Finally,  a  large  problem  that  will  have  to  be  overcome  is  that  many  of  the  variables 
most  useful  for  process  measurements  may,  depending  on  how  SynTEAM  is  finally 
implemented,  be  spoken  commands.  For  example,  passing  information  when  needed  is  a 
good  indicator  of  an  efficient  team,  and  since  WDs  do  this  verbally  there  may  be  no  easy 
way  for  the  system  to  capture  this  (i.e.,  it  may  have  to  be  done  the  hard  way  -  through 
transcribing  the  voice  data). 


CONCLUSIONS 


We  note  that  our  CTA  is  obviously  not  comprehensive.  We  did  not  sample  all  the 
specific  events  that  could  occur  in  the  WD  task  (and/or  describe  fully  how  WDs  dealt  with 
the  events  we  did  catalog).  Nor  did  we  capture  all  the  strategies  and  tactics  used  by  WDs. 
To  some  extent  a  CTA  that  approaches  this  kind  of  comprehensiveness  may  only  be 
possible  after  extensive  observation  of  WDs  within  an  existing  synthetic  task  environment 
(e  g.  a  simulator),  and  then  only  under  a  diverse  set  of  scenarios  meant  to  fiilly  elicit  their 
job  expertise.  Another  issue  is  that  while  the  higher-level  strategic  decisions  were 
especially  important  in  our  characterization  of  a  synthetic  WD  task,  in  reality  WDs  do  not 
ordinarily  exercise  this  authority,  deferring  to  the  SD.  This  represents  a  limitation  on  the 
population  we  interviewed;  however,  this  population  did  include  some  SDs. 

The  current  report  does  not  exhaust  our  CTA  data  (or  ideas),  and  there  may  be 
other  ways  to  mine  the  data  to  further  inform  SynTEAM  implementation  decisions. 
However,  our  CTA  data,  as  it  stands,  produced  a  good  general  overview  of  the  WD  task. 
Inherent  in  the  summaries  of  the  CTA  data  we  hope  to  have  shown  a  good  starting  point 
for  the  development  of  an  ecologically  valid  synthetic  AW  ACS  task. 
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APPENDICES 


Appendix  A:  Interview  Guides 

N.B.;  In  Sections  A  and  E,  only  those  probes  written  in  italics  are  intended  to  be 
used  verbatim.  In  Sections  B,  C,  and  D,  all  probes  can  be  used  verbatim. 

---A:  General  Interview  Guide 

1 .  Ask  WD  for  background  information 

-  Rank,  years  in  the  Air  Force. 

-  Years  as  a  WD. 

-  Experience  as  an  SD? 

-  Variety  of  missions  flown. 

2.  Ask  WD  to  describe  the  job  characteristics  and  list  a  WD’s  main  tasks/responsibilities; 

-  Emphasis  on  concrete  examples. 

-  Which  responsibilities/tasks  place  the  highest  burden  on  a  WD? 

-  Which  aspects  of  the  job  are  cognitively  challenging? 

*  E.g.,  tasks  that  task  memory,  attention,  impose  a  high  workload. 

-  Knowledge  Audit;  What  are  the  main  skills  of  a  WD? 

*  Past  and  future  projection: 

“Have  you  ever  walked  into  a  situation  and  known  exactly  ‘what  the  score  was '?  ’’ 

*  The  “big  picture”  view; 

“What  is  important  about  the  Big  Picture?  What  do  you  need  to  keep  track  of”? 

*  Noticing  cues: 

“Have  you  had  experiences  where  part  of  a  situation  'popped  out '  at  you? 

Did  you  notice  things  others  didn ’t  catch?  Give  an  example  ” 

*  Job  smarts: 

“Do  you  use  any  strategies  that  allow  you  to  work  smart  or  accomplish  more?  ” 

*  Opportunities/improvising: 

“Give  an  example  where  you  have  had  to  improvise  or  find  a  better  method  ” 

*  Anomalies: 

“Give  examples  of  when  you  spotted  a  deviation  from  the  norm,  or  knew 
something  was  wrong.  ” 

*  Equipment  difficulties: 

“Are  you  ever  led  astray  by  the  equipment?  How  do  you  ‘trade  off’  between 
what  the  equipment  says  and  your  own  judgment?  ” 

3 .  Ask  WD  to  list  the  different  kinds  of  missions  in  which  he/she  may  be  called  upon  to 
participate  (e  g.,  defensive  counter  air,  strike,  search  and  rescue): 

-  Give  concrete  examples  of  each  kind  of  mission. 

*  Doctrine,  Rules  of  Engagement. 

-  Discuss  the  difficulties  encountered  on  each  kind  of  mission. 
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*  Cognitive  demands  (memory,  attention,  workload,  etc  ). 

*  Number  of  decisions. 

*  “Tempo”  of  mission. 

*  Degree  of  teamwork  (communication,  cooperation). 

-  Discuss  the  different  strategies  used  in  each  type  of  mission. 

*  What  are  the  advantages  and  disadvantages  of  each  strategy? 

*  Degree  of  teamwork  (communication,  cooperation). 

4.  Work  through  a  series  of  “generic”  missions/strategies  (defensive  counter  air,  strike, 
search  and  rescue); 

-  List  important  decision  points,  with  an  emphasis  on  concrete  examples. 

-  Discuss  the  information  that  feeds  into  each  decision. 

*  How  is  this  information  obtained  (e  g.,  pattern  matching  on  the  screen)? 

*  Is  it  easy  to  obtain  this  information? 

-  Discuss  which  decisions  require  the  most  teamwork  (communication,  cooperation). 

*  Which  decisions  that  require  input  from  other  team  members? 

*  Of  which  decisions  do  other  team  members  need  to  be  notified? 

*  How  is  information  shared  between  team  members? 

*  What  are  the  impediments  to  sharing  information? 

-  How  important  is  shared  knowledge? 

*  Mental  model  of  the  job  (degree  of  overlap). 

*  Mental  model  of  the  mission  (degree  of  overlap). 

*  Situation  awareness  of  the  other  WDs’  situations. 

-  Which  decisions  are  most  cognitively  complex  or  difficult? 

*  What  is  cognitively  complex  about  it? 

*  What  cognitive  demands  are  made  (perception,  memory,  etc.)? 

-  “Scenario  from  hell”: 

*  “If you  were  going  to  devise  a  scenario  to  show  someone  what  this  job  is  really 

about,  what  would  you  put  in  it?  ” 

5.  Discuss  current  AW ACS  training; 

-  How  does  training  address  the  needs  of  the  job? 

-  How  does  training  prepare  WDs  for  the  more  demanding  aspects  of  the  job? 

-  Does  the  training  address  teamwork  and  communication? 

-  How  could  training  be  improved? 

— B:  Critical  Incident  Interview  Guide 

1.  Get  critical  incident: 

-  “Relate  to  me  an  incident  that,  in  your  experience,  required  a  lot  of  teamwork  and 
challenged  your /the  team 's  skills.  ’’ 

2.  Have  WD  construct  a  timeline  of  this  critical  incident  (flow  of  events) 

3 .  Review  the  timeline  with  the  WD,  filling  in  the  gaps  and  adding  other  things  that  come 
to  mind 
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4.  Focusing  on  events  on  the  timeline,  ask  about: 

•  WD’s  focus  of  attention: 

-  how  and  why  it  changed  as  incident  progressed. 

-  were  changes  good  or  bad? 

•  cues  attended  to: 

-  determining  degree  of  enemy  threat  and  status  of  friendly  resources. 

-  reliance  on  information  in  other  WDs’  lanes,  or  information  from  other  WDs. 

•  communication/cooperation  with  other  team  members: 

-  when  and  how  communication/cooperation  took  place. 

-  was  communication/cooperation  successful? 

-  modes  of  communication/cooperation. 

-  communication  that  should  have  occurred. 

•  options  considered: 

-  reasons  for  selecting  the  final  course  of  action. 

•  resource  allocation: 

-  why  WD  selected  particular  tracks  to  commit  against  the  enemy. 

•  loss/deterioration  of  situation  awareness: 

-  means  of  regaining  S  A. 

-  help  from  other  team-members  in  regaining  SA. 

— C:  Quasi-Performance  Interview  Guide 

The  bulleted  text  represents  the  actual  questions,  while  the  headings  in  square 
brackets  are  meant  to  remind  interviewers  of  the  PARI  categories. 

Situation  Awareness  Probes:  “What’s  going  on  now.” 

•  Give  me  an  overview  of  what  is  going  on,  both  in  WD2’s  and  the  other  WDs’  lanes. 

•  What  are  the  current  rules  of  engagement?  Explain  the  rules  of  engagement. 

•  What  resources  does  WD2  control?  Draw  them  on  the  map,  along  with  their  HAS*. 

•  What  targets  are  in  WD2’s  area?  Draw  them  on  the  map,  along  with  their  HAS*. 

•  What  are  WD2’s  resources  doing  and  why? 

•  How  would  you  prioritize  the  targets  in  WD2’s  lane?  Do  you  think  WD2  is 
prioritizing  them  differently?  How?  Why? 

•  What  resources  do  the  other  WDs  control?  Draw  them  on  the  map,  along  with  their 
HAS*. 

•  What  targets  are  in  the  other  WDs’  area?  Draw  them  on  the  map,  along  with  their 
HAS*. 

•  What  are  the  other  WDs’  resources  doing  and  why? 


•  Do  you  think  that  the  situation  is  under  control?  Why? 

•  Talk  about  how  the  WDs  work  as  a  team.  Do  they  communicate  well,  etc  ? 

•  What  do  you  think  will  happen  in  the  next  few  minutes? 

•  Be  sure  to  emphasize  that  we  would  like  the  WDs  to  draw  any  information  on  the  map, 
no  matter  how  sketchy  (if  this  is  not  emphasized  they  might  not  draw  aircraft  unless  they 
remember  the  position,  HAS  AND  the  call  sign).  If  interviewee  does  not  at  least  know 
details  of  WD2’s  assets,  they  are  not  “mission  ready”  and  so  discreetly  switch  to  a  general 
interview  format. 

“Next  Action”  Probes:  “Put  yourselves  into  the  situation.  What  would  you  do?” 
[ACTION] 

•  What  would  you  do  in  the  next  minute  or  so? 

[PRECURSOR] 

•  Why  would  you  perform  these  actions?  What  would  your  goals  be? 

•  What  information  would  influence  your  decisions?  Why? 

•  Where  and  how  would  you  get  this  information?  Would  you  get  it  from  other  team- 
members? 

•  Would  you  be  influenced  by  anything  going  on  in  the  other  WDs’  lanes? 

•  Discuss  how  target  priorities  and  positions  (discussed  in  the  last  section)  would  affect 
what  you  decided  to  do. 

[RESULTS] 

•  Project  what  the  results  of  your  actions  would  be. 

•  Would  your  actions  impact  the  other  WDs?  Would  you  need  to  convey  information  to 
anyone  else  as  a  result  of  his  actions?  Who?  What?  Why?  How? 

[INTERPRETATION] 

•  How  would  your  actions  change  your  overall  situation  model/mission  objectives? 
Would  your  actions  change  your  prioritization  of  the  targets  in  WD2’s  lane  (eg.. 
Tactical  Sort)?  How? 

[ALTERNATIVE  ACTIONS] 

•  Given  your  current  goals/priorities,  discuss  how  you  could  meet  your  goals  via 
different  courses  of  action.  Include  any  alternative  actions  you  could  take,  or  different 
sequences  in  which  actions  could  be  performed. 

•  Why  wouldn’t  you  take  these  courses  of  action? 
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•  Discuss  how  having  different  information  would  affect  how  you  would  decide  to  reach 
your  goal.  Where  would  you  get  this  information?  Would  you  get  it  from  other  team 
members? 

[ALTERNATIVE  PRECURSORS] 

•  What  other  goals  could  you  have  at  this  point?  Discuss  how  your  goals  would  have 
changed  if  you  had  prioritized  targets  differently. 

•  Why  did  you  choose  not  to  pursue  these  goals/use  this  prioritization? 

•  Discuss  how  having  different  information  would  have  affected  your  goals  at  this  point. 
Where  would  you  get  this  information?  Would  you  get  it  from  other  team  members? 

[ALTERNATIVE  RESULTS/INTERPRETATION] 

•  What  else  could  happen  as  the  results  of  your  actions  (e.g.,  if  “something  goes 
wrong)?  What  would  this  mean? 

“Action  Taken”  Probes:  “Answer  questions  on  events  that  occurred  in  the  last  tape 

segment.” 

[ACTIONS] 

•  What  (external)  events  took  place  in  the  last  tape  segment?  Please  answer  in  order  of 
importance  (Planes  shot  down,  new  planes,  etc.). 

•  What  actions/decisions  did  WD2  take? 

•  Did  WD2  receive  any  information  from  WDl  or  WD3  during  the  last  tape  segment? 
Was  this  information  accurate? 

•  Did  anything  else  of  consequence  happen  during  the  last  tape  segment? 

[PRECURSORS] 

•  Why  do  you  think  WD2  performed  the  actions  he  did?  What  goals  was  he  trydng  to 
accomplish? 

•  What  information  do  you  think  influenced  WD2’s  decisions?  Why? 

•  Where  and  how  did  WD2  get  this  information?  Did  he  get  it  from  other  WDs?  Was  the 
information  he  received  accurate? 

•  Do  you  think  that  WD2  was  influenced  by  anything  going  on  in  the  other  WDs’  lanes? 

•  Discuss  how  target  priorities  and  positions  might  have  affected  what  WD2  did. 

[RESULTS] 

•  What  were  the  results  of  WD2’s  actions? 

•  Did  WD2’s  actions  impact  the  other  WDs?  Did  the  WD  need  to  convey  information  to 
anyone  else  as  a  result  of  his  actions?  Who?  What?  Why?  How? 
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[INTERPRETATION] 

•  Discuss  how  actions/events  in  the  last  segment  might  have  changed  WD2’s  overall 
situation  model/mission  objectives.  How?  Discuss  how  you  would  change  your 
priorities  (e  g.,  Tactical  Sort). 

[ALTERNATIVE  ACTIONS] 

•  Given  WD2’s  (inferred)  goals/priorities,  discuss  any  other  actions  he  could  have  taken 
at  this  point  that  would  have  satisfied  these  goals. 

•  Why  do  you  think  he  didn’t  take  these  actions? 

•  Discuss  how  having  different  information  might  have  affected  which  actions  WD2 
decided  to  take  at  this  point.  Where  would  he  have  gotten  this  information?  Would  he 
have  gotten  it  from  other  team  members? 

•  Did  WD2  do  all  he  needed  to  do?  If  not,  why  not? 

•  How  did  WD2’s  actions  differ  from  your  planned  actions?  Did  he  use  different 
information  to  make  decisions?  How?  Why? 

[ALTERNATIVE  PRECURSORS] 

•  How  did  WD2’s  (inferred)  goals  differ  from  your  goals?  Do  you  think  he  used 
different  information  to  set  goals/priorities?  How?  Why? 

[ALTERNATIVE  RESULTS/INTERPRETATION] 

•  What  else  could  have  happened  in  the  last  segment?  What  would  this  have  meant? 
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Appendix  B:  Critical  Incidents 

Critical  incidents  are  written  in  the  first  person,  as  they  were  recounted  to  us. 
However,  in  the  interests  of  clarity  the  raw  reports  have  been  paraphrased.  Interviewer 
conunents  have  been  omitted  from  the  bodies  of  the  reports,  although  an  interviewer 
commentary  has  been  added  at  the  end  of  each  incident.  Figures  are  copies  of  explanatory 
drawings  made  by  interviewees. 

—  1)  A  Difficult  Air  Refueling 

This  particular  day  we  had  three  tankers  up  there,  and  I  was  in  charge  of  refijeling 
all  the  strike  aircraft.  It  was  a  rainy  day  and  pilots  were  having  trouble  with  visibility  due 
to  the  clouds,  so  I  was  giving  the  aircraft  more  fuel  to  make  sure  they  could  get  back.  In 
the  middle  of  refueling  the  strike  package,  I  received  an  emergency  call  from  a  British 
Jaguar.  He  was  screaming  for  fuel.  His  trouble  was  made  worse  because  Jagyars  don’t 
have  radar. 

I  quickly  started  calculating  which  tanker  I  could  vector  out  to  the  Jaguar.  I 
needed  to  check  how  many  aircraft  were  on  each  tanker  and  also  the  fuel  status  of  each 
tanker  (to  ensure  that  the  tanker  had  enough  fuel  to  get  out  to  the  Jaguar,  fuel  him,  and 
return  back  to  the  strike  team).  Before  I  had  time  to  decide,  one  of  the  tankers,  who  had 
been  listening  to  the  radio,  responded  that  he  could  go.  This  was  a  big  help  in  making  my 
decision,  and  it  was  an  advantage  that  the  tanker  pilot  knew  what  was  going  on.  I  then 
coordinated  with  the  tanker  to  help  the  Jaguar. 

The  SD  didn’t  get  too  involved  in  this  situation,  because  I  was  constantly  thinking 
ahead  and  presenting  him  with  a  good  solution  at  each  stage  of  the  problem.  In  the  end  I 
decided  to  do  a  150/30  rendezvous.  That  is  where  you  have  the  tanker  and  the  other 
aircraft  fly  toward  one  another,  then  as  they  near  each  other  the  tanker  does  a  150  degree 
turn  and  the  other  aircraft  does  a  30  degree  turn,  rolling  right  in  behind  the  tanker  (see 
Figure  El).  The  key  here  is  getting  the  fighter  to  do  the  least  amount  of  turning,  in  order 
to  save  fuel. 

The  main  points  within  this  incident  were  deciding  which  tanker  to  use  and 
figuring  out  how  to  get  the  aircraft  together  with  minimal  fuel  loss,  all  the  while  keeping 
them  out  of  the  wrong  air  space.  After  the  refueling,  I  pushed  the  Jaguar  back  to  the 
Check-in  Controller  to  send  him  home,  and  vectored  the  tanker  back  to  refuel  the  strike 
package.  While  calculating  all  of  that,  I  still  had  to  manage  the  refueling  with  the  two 
other  tankers:  first,  I  would  make  sure  that  the  intercept  between  the  Jaguar  and  the 
tanker  was  going  well  (emergencies  are  always  top  priority),  while  still  monitoring  the 
other  tankers. 
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Figure  El:  A  150/30  Rendezvous 
Commentary 

Weather  was  a  major  factor  here.  The  Jaguar  had  used  a  lot  of  fuel  flying  around 
looking  for  clear  spaces  in  the  clouds,  which  is  what  had  gotten  him  into  trouble.  The  WD 
suddenly  had  to  decide  which  tanker  to  use,  requiring  him  to  check  each  tanker’s  refueling 
schedule  and  try  to  form  an  impromptu  solution  while  still  monitoring  the  other  refuelings 
taking  place.  Since  the  WD  was  controlling  refueling  of  a  large  strike  team  on  three 
tankers,  this  was  not  easy.  Fortunately,  he  received  help  from  the  tanker  pilot,  who  was 
listening  to  the  chatter  on  the  tanker  fi-equency.  The  tanker  pilot  came  up  with  his  own 
solution,  considerably  reducing  the  burden  on  the  WD.  Once  the  WD  decided  to  go  with 
the  tanker  pilot’s  plan,  he  had  to  calculate  some  very  complex  and  precise  vectoring 
geometry  while  still  overseeing  the  refueling  on  the  other  two  tankers.  This  required  him 
to  switch  rapidly  between  two  scope  views  in  order  to  keep  track  of  all  that  was  going  on. 
This  incident  highlights  how  WDs  have  to  form  rapid  solutions  to  ambiguous  problems, 
deal  with  more  than  one  task  at  one  time,  and  shows  how  communication  between  the 
WDs  and  pilots  can  expedite  a  solution. 

—  2)  A  Potential  “Friendly  Fire”  Incident  with  a  Helicopter 

This  is  not  an  elaborate  example,  but  it  scared  me.  I  was  controlling  some  fighters 
patrolling  an  fiiendly  airspace  next  to  hostile  territory  (see  Figure  E2).  I  was  the  AOR 
controller  in  this  mission.  The  weather  was  bad  and  the  visibility  wasn’t  very  good,  so  a 
lot  of  the  aircraft  were  flying  in  and  out  of  the  clouds  trying  to  get  above  them.  That 
makes  our  job  a  lot  harder,  because  the  aircraft  often  get  lost  out  there  and  need  our  help. 
There  was  a  UN  helicopter  scheduled  to  fly  across  the  fnendly  territory  into  the  hostile 
territory  (above  the  dotted  horizontal  line  in  Figure  E2).  We  knew  exactly  who  he  was, 
and  we  had  his  flight  plan.  He  took  off  and  was  flying  across  the  “No  Fly”  zone.  At  about 
the  same  time,  we  picked  up  an  “unknown”  track  in  the  hostile  airspace.  I  personally  did 
not  see  him,  because  I  was  talking  to  the  FI  5s,  and  it  was  also  my  job  to  track  the  UN 
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helicopter.  Tracking  helicopters  is  a  tough  job,  because  they  are  low  and  slow  and  often 
drop  off  the  scope.  I  heard  about  the  unknown  from  one  of  the  other  WDs,  who  pointed 
to  it  on  my  scope. 

The  unknown  worried  me,  so  I  committed  two  F-15s  (on  the  left  of  Figure  E2)  to 
check  it  out.  As  the  F- 15s  were  on  their  way  toward  the  unknown,  the  unknown  turned 
away  and  headed  back  north.  I  did  not  notice  this,  though.  The  F-15s  locked  on  to  the 
helicopter  instead  of  the  unknown,  because  it  was  in  the  area  where  the  unknown  was 
projected  to  be,  and  the  weather  was  bad.  Things  happened  real  fast.  One  of  the  WDs 
yelled  at  me,  “He  (the  bad  guy)  turned  north,  pull  them  (F15s)  off!”,  which  I  did. 


Figure  E2.  Potential  friendly  fire  with  a  helicopter. 

The  bad  weather  was  a  major  factor  in  this  incident;  the  F-15  pilots  didn’t  like  not 
being  able  to  actually  see  what  they  were  conunitted  on,  so  they  got  too  close  to  the 
helicopter.  This  was  a  big  potential  problem.  Afterwards,  we  concluded  that  the  unknown 
was  probably  either  an  enemy  helicopter  or  fighter  that  had  come  down  to  investigate  the 
UN  helicopter.  They  knew  that  the  UN  helicopter  was  coming,  so  they  wouldn’t  have  shot 
it  down. 

This  incident  is  a  good  example  of  my  team  helping  me  out.  I  was  so  “tunneled  in” 
on  the  helicopter  and  the  F-15s  that  I  completely  missed  the  unknown.  The  other  WDs 
helped  out  with  tracking  and  giving  updates  on  the  unknown.  In  this  case,  we  had  proper 
mission  planning  and  had  flown  together  a  lot,  so  that  helped. 
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Commentary 

Again,  bad  weather  was  a  factor.  The  WD  was  quite  overloaded,  monitoring  three 
tracks  (two  sets  of  F-15s  and  a  helicopter)  in  different  areas  of  the  scope.  Helicopters  are 
notoriously  difficult  to  track,  since  they  fly  low  and  slowly,  often  disappearing  from  radar. 
The  WD  became  “tunneled  in”  on  the  F-15s  he  had  committed,  and  he  did  not  see  that  the 
unknown  had  turned  (and  he  also  forgot  about  the  helicopter).  This  incident  shows  how 
WDs  can  become  overloaded  when  they  have  to  perform  too  many  tasks,  especially  in 
different  parts  of  the  scope,  requiring  frequent  scope  adjustment,  and  when  one  of  the 
tasks  (tracking  the  helicopter)  has  to  be  done  with  imperfect  information.  It  highlights  the 
importance  of  WDs  having  a  common  situation  model;  the  other  WD  was  able  both  to  tell 
the  protagonist  that  there  was  a  hostile  aircraft  in  the  area,  and  to  help  when  he  became 
tunneled  in  and  did  not  notice  that  the  hostile  had  turned  away  and  that  the  F-15s  were  in 
fact  locked  on  to  the  helicopter. 

—  3)  A  Potential  “Friendly  Fire”  Incident  with  Fighters 

I  was  the  AOR  controller  in  this  incident,  in  the  middle  seat,  and  there  was  a  HVA 
Controller  and  a  Check-in  Controller.  A  Canadian  EFl-1 1  came  off  his  tanker,  and  didn’t 
check  in  with  us  when  he  came  back  into  the  “box”  (the  AOR),  so  we  didn’t  know  he  was 
there.  All  of  a  sudden,  some  of  our  F-16s  said  “contact”,  meaning  that  they’d  picked  up 
something  on  their  radars,  and  giving  bearing  and  range  off  their  noses  -  that  is,  relative  to 
their  own  aircraft.  However,  I  saw  nothing  on  the  radar  screen  in  the  area  that  they  were 
indicating.  I  asked  the  HVA  Controller  if  he  had  anything  on  his  scope,  in  case  my  scope 
was  malfimctioning.  He  also  saw  nothing  on  his  scope.  Meanwhile,  the  F-16s  were  getting 
more  agitated  as  they  closed  in.  The  SD  came  over  and  was  looking  over  my  shoulder.  I 
asked  the  F-16s  to  give  me  the  bearing  and  range  of  the  target  relative  to  the  bullseye,  to 
get  a  non-egocentric  frame  of  reference,  but  they  continued  to  give  bearing  and  range  off 
their  noses.  The  SD  wanted  to  follow  the  ROE  and  get  a  VIS  ID  (Visual  Identification). 
The  F-16s  were  closing  in  fast,  but  finally  they  visually  identified  some  EFl-1  Is.  The  HVA 
Controller  had  thought  that  the  EFl-1  Is  were  still  on  tanker,  so  he  checked  with  the 
tanker  and  found  out  that  this  was  not  the  case.  It  must  have  come  off  the  tanker  and  had 
not  checked  in  with  us.  He  let  me  know,  and  I  called  off  the  F-16s,  just  in  time. 

If  the  F-16  pilots  had  given  contact  off  the  bullseye,  the  EFl-1  Is  would  have  heard 
it  and  realized  that  they  were  in  danger.  People  weren’t  communicating  well.  After  the 
incident  we  had  a  big  briefing  on  the  ground  with  the  fighter  pilots,  telling  the  FI 6s  that 
they  needed  to  follow  requests,  and  the  EFl-1  Is  that  they  needed  to  check  in  when  they 
came  back  in  the  box. 

Commentary 

There  was  a  major  problem  with  information  here  -  the  target  onto  which  the 
fighters  locked  was  not  on  the  WD’s  scope,  so  the  WD  was  working  with  limited 
information,  under  time  pressure.  The  incident  shows  how  a  number  of  small 
miscommunications  -  the  EFl-1  Is  not  checking  back  into  the  box,  the  refusal  to  cooperate 
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on  the  part  of  the  F-16  pilots  -  can  compound  into  a  major  problem.  However,  the  incident 
also  shows  how  communication  between  WDs  can  expedite  a  solution:  the  AOR  WD  was 
able  to  concentrate  on  the  fighters  while  the  HVA  WD  communicated  with  the  tankers  to 
get  the  necessary  information. 

—  4)  A  Critical  Incident  Related  to  the  ‘‘Black  Hawk  Shoot  Down” 

In  this  mission,  we  encountered  a  helicopter  that  wasn’t  on  the  ATO.  The  ATO  is 
a  break-down  of  all  the  aircraft  that  are  going  to  be  up  that  day,  including  their  schedules, 
and  the  modes  and  codes  that  they  will  be  transmitting.  We  need  to  know  all  this 
information,  because  we  are  the  command  and  control  platform  for  that  area.  The  ATO 
procedures  are  set  and  we  follow  them  exactly. 

However,  helicopter  pilots  tend  not  to  follow  the  ATO.  Helicopters  are  often 
owned  by  the  Army,  and  there  is  a  big  disconnect  between  the  Air  Force  and  the  Army. 

For  example,  we  would  have  helicopters  take  off  from  Zakho,  and  some  would  check  in, 
but  some  wouldn’t.  Sometimes  they  would  squawk  Mode  4,  sometimes  they  wouldn’t; 
rarely  would  they  have  the  right  Mode  1,  which  is  a  big  ID  code  for  us  on  the  AW  ACS.  If 
they  did  check  up  they  would  say,  for  example,  “This  is  Black  Hawk  21  inbound  to  Arbil,” 
and  that  was  all  we  would  hear  from  them.  The  big  thing  is  that  we  knew  who  the 
helicopters  were,  we  just  didn’t  know  how  they  fit  into  the  game  plan,  and  we  really  do 
need  to  know  this.  In  this  incident,  we  were  able  to  the  recognize  the  helicopter  problem 
from  the  outset,  probably  because  we  had  more  experience  going  in,  and  we  had 
developed  some  pretty  good  procedures  for  following  helicopters. 

The  AW ACS  has  a  hard  time  tracking  helicopters  because  they  are  low  and  slow: 
you  may  pick  them  up  every  now  and  then,  but  sometimes  cannot.  You  primarily  have  to 
track  them  off  of  IFF,  not  radar.  It  is  very  easy  for  symbology  to  drift  off  the  radar  return. 
These  guys  usually  only  check  up  once,  if  they  bother  to  check  up  at  all,  and  it  is  easy  to 
forget  about  them.  If  we  don’t  expect  someone  up  there,  or  the  helicopter  is  not  on  the 
ATO,  he  is  forgotten. 

The  In-Route  Controller  picked  up  a  helicopter  on  the  radio.  The  helicopter 
identified  itself,  but  used  a  call  sign  that  was  not  on  the  ATO  or  the  Master  ATO. 
However,  based  on  where  it  was  coming  from,  we  guessed  it  was  an  Army  helicopter.  If 
the  In-Route  Controller  has  experience,  or  is  briefed  that  such  an  incident  may  happen,  he 
will  tag  him  up.  However,  in  this  incident  the  In-Route  Controller  had  no  idea  where  to 
look.  We  (the  other  WDs)  were  not  doing  anything,  so  we  were  helping  him.  The  In- 
Route  Controller  didn’t  know  where  the  Army  base  was,  so  I  reached  over  and  pointed  on 
his  scope.  He  started  tracking,  but  only  got  intermittent  blips  on  the  scope.  He  tracked  him 
to  the  box,  at  which  point  the  helicopter  should  have  checked  in  and  got  put  on  the  AOR 
frequency.  However,  he  didn’t  check  in,  and  so  did  not  get  put  on  the  AOR  frequency.  My 
F-15s  came  in  and  swept  the  area.  They  were  not  expecting  anyone  in  the  area,  but  they 
picked  up  on  a  helicopter  out  there.  I  knew  that  there  were  fiiendly  helicopters  in  the  area, 
but  we  didn’t  know  what  the  Army  was  doing  out  there.  The  F-15s  moved  in,  locked  on 
to  the  helicopter,  and  radioed  to  us.  I  could  not  see  the  helicopter,  so  I  asked  the  In-Route 
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Controller  if  he  knew  where  the  helicopter  he  was  tracking  was.  At  that  point,  the  In- 
Route  Controller  noticed  that  the  helicopter  had  passed  Gate  One  going  into  the  box  and 
needed  to  be  put  on  the  AOR  frequency.  That  is  frequency  coordination.  If  no  one  had 
caught  this,  the  helicopter  would  have  stayed  on  the  In-Route  Controller’s  frequency  until 
he  reached  his  destination.  This  would  have  been  a  bad  thing,  because  I  was  controlling  all 
the  fighters  out  there,  not  the  In-Route  Controller,  and  I  wouldn’t  have  been  able  to 
contact  the  helicopter  on  my  frequency,  so  I  wouldn’t  have  known  who  he  was. 

Commentary 

Again,  this  incident  was  based  on  miscommunication,  not  so  much  between  the 
WDs  but  more  between  the  helicopter  pilots  and  the  WDs.  The  In-Route  controller  was 
working  with  imperfect  information:  the  same  problem  with  helicopters  periodically 
dropping  from  the  radar.  Coordination  between  the  WDs  saved  this  from  becoming  a 
major  incident:  if  WDs  didn’t  have  the  same  mental/situation  model,  they  would  not  have 
picked  up  on  the  In-Route  controller’s  error  of  not  putting  the  helicopter  on  the  AOR 
frequency. 

—  5)  Refueling  the  E3  AWACS  Aircraft 

One  time  the  E3  was  refueling,  which  means  that  the  radar  was  shut  down  and 
“other  agencies”  were  controlling  the  aircraft  that  were  up  (including  us).  When  our  radar 
came  back  up,  the  ASO  did  a  diagnostics  on  the  equipment  and  declared  the  radar 
operational,  as  usually  happens.  Once  we’re  back  online  we’re  responsible  for  the  area. 
However,  all  of  a  sudden  we  were  faced  with  6  hostile  aircraft  flying,  and  we  had  no  other 
aircraft  up.  In  this  case  we  had  no  option  but  to  turn  and  run.  The  MCC  got  on  the  radio, 
but  the  hostiles  all  landed  soon  after,  so  nothing  happened.  If  there  had  been  assets 
airborne  and  they  were  being  controlled  by  the  other  agency,  we  would  have  coordinated 
with  that  agency  to  take  over,  maybe  wait  for  a  lull  in  the  action.  The  SD  usually  does  the 
coordination  with  the  other  agencies.  The  general  feeling  was  that  the  AWACS  schedule 
was  getting  too  predictable,  and  we  needed  to  change  this  in  order  to  not  be  so  vulnerable. 

Commentary 

The  main  point  of  this  incident  is  that  the  WD  environment  is  both  dynamic  and 
uncertain:  the  WDs  were  thrown  into  a  situation  where  things  were  suddenly  different 
from  before,  and  they  had  to  work  together  quickly  to  address  the  new  situation. 

—  6)  A  Stateside  Incident:  Large  Force  Exercise  in  Utah 

The  day  of  the  incident  I  was  controlling  2  F4s  flying  together.  One  of  my  F4s  lost 
an  engine,  but  did  not  tell  me  that  he  was  in  trouble.  I  had  to  infer  what  was  going  on.  The 
pilot  said,  “stand  by  one,  can  you  get  the  wing  man  over?”  As  soon  as  he  said  that  I 
recognized,  from  experience,  that  something  was  up.  So  I  vectored  the  wing-man  over 
and  informed  the  SD  that  I  thought  something  was  wrong.  The  wing-man  went  over  there 
and  checked  out  the  plane  for  structural  damage,  and  relayed  to  the  pilot  that  he  had  lost 
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only  one  engine:  the  other  was  still  operational,  so  he  could  fly.  However,  the  loss  of  an 
engine  means  a  loss  in  maneuverability,  and  he  was  also  low  on  fiiel.  The  WD  sitting  next 
to  me  picked  up  on  the  situation,  and  took  over  talking  to  ATC  to  help  out  so  I  could  then 
just  talk  to  the  pilot.  We  had  to  work  together:  it  was  three-way  teamwork  between 
myself,  the  controller  next  to  me,  and  the  SD  to  get  pilot  back  to  base. 

The  key  to  any  emergency  is  to  listen,  because  you  want  to  hear  the  pilot  going 
down  his  check  list,  trying  to  figure  out  what  is  wrong.  You  don’t  want  to  be  talking  on 
the  radio  and  interrupting  his  thoughts  when  assessing  the  situation.  If  there  is  something 
that  I  need  that  the  pilot  is  not  giving  me,  then  I  would  ask.  The  center  WD  was  helping 
me  pick  up  information  from  what  the  pilot  was  saying  to  me,  then  passing  that 
information  on  to  ATC  so  they  could  be  prepared  handle  the  incoming  emergency. 

Commentary 

This  incident  emphasizes  the  importance  of  communication  and  teamwork.  Even 
though  the  WD  was  not  directly  communicating  with  the  pilot,  the  fact  that  the  WD  could 
hear  the  pilot  doing  his  checklist  was  obviously  a  great  help  here.  The  fact  that  the  WD 
was  able  to  quickly  and  easily  pass  off  the  task  of  coordinating  with  ATC  to  another  WD 
greatly  helped  him  to  concentrate  on  the  task  at  hand. 

—  7)  A  “Downed”  Aircraft 

Once,  while  controlling  over  Michigan,  ATC  told  us  that  a  military  aircraft  had 
gone  down.  All  WDs  immediately  counted  their  own  aircraft.  All  WDs  were  experienced. 
The  SD  said  to  stop  whatever  we  were  doing,  and  all  planes  went  to  CAP.  I  became  the 
coordinator  between  ATC  and  the  WDs,  known  as  a  “pit  boss”.  The  SD  was  doing  the 
actual  coordination,  I  was  relajdng  information  to  the  WDs,  like  a  2"‘‘  SD.  We  used  our 
own  aircraft,  and  sent  some  FI 6s  to  look  over  the  exact  area.  We  saw  nothing,  but 
continued  to  look.  The  fighters  were  eventually  sent  home,  although  we  stayed  out.  The 
SD  did  a  lot  of  coordination  with  the  ground.  Finally,  all  was  explained  and  we  went 
home:  a  C130  had  gone  low  over  a  ridge,  and  a  witness  had  thought  it  had  crashed. 

Commentary 

This  incident  illustrates  how  new  situations  can  quickly  unfold,  and  how  WDs 
sometimes  need  to  coordinate  and  reconfigure  their  teams  quickly  to  deal  with  new 
problems. 
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